[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:45 dist-test-slave-5vc0 krb5kdc[20021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:45 dist-test-slave-5vc0 krb5kdc[20021](info): set up 2 sockets
Sep 28 02:50:45 dist-test-slave-5vc0 krb5kdc[20021](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:50:46.660894 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:50:46 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:50:46.777199 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:46869
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:46869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:46.886590 20034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:46.886880 20034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:46.886940 20034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:46.891016 20034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:46.891096 20034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:46.891124 20034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:46.891144 20034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:46.891165 20034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:46.896710 20034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:46869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:46869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:46.897850 20034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:46.898885 20034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:46.907789 20034 server_base.cc:1034] running on GCE node
W20260928 02:50:46.907804 20040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:46.907881 20039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:46.908048 20042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:46.908643 20034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:46.909880 20034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:46.911044 20034 hybrid_clock.cc:648] HybridClock initialized: now 1790563846911028 us; error 32 us; skew 500 ppm
Sep 28 02:50:46 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563846, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:46.914001 20034 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:50:46.914924 20034 webserver.cc:460] Webserver started at http://127.19.139.190:39301/ using document root <none> and password file <none>
I20260928 02:50:46.915467 20034 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:46.915534 20034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:46.915738 20034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:46.917865 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "e1cba0b02847427e895d45bc7eb94b45"
format_stamp: "Formatted at 2026-09-28 02:50:46 on dist-test-slave-5vc0"
server_key: "4267a2c7931b5e4c682c1ea90b0372a8"
server_key_iv: "f2bdbe702ad0521911f95096c5b5c2f1"
server_key_version: "encryptionkey@0"
I20260928 02:50:46.918419 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "e1cba0b02847427e895d45bc7eb94b45"
format_stamp: "Formatted at 2026-09-28 02:50:46 on dist-test-slave-5vc0"
server_key: "4267a2c7931b5e4c682c1ea90b0372a8"
server_key_iv: "f2bdbe702ad0521911f95096c5b5c2f1"
server_key_version: "encryptionkey@0"
I20260928 02:50:46.922276 20034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:50:46.924784 20049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:46.926084 20034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:46.926203 20034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "e1cba0b02847427e895d45bc7eb94b45"
format_stamp: "Formatted at 2026-09-28 02:50:46 on dist-test-slave-5vc0"
server_key: "4267a2c7931b5e4c682c1ea90b0372a8"
server_key_iv: "f2bdbe702ad0521911f95096c5b5c2f1"
server_key_version: "encryptionkey@0"
I20260928 02:50:46.926311 20034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:46.938452 20034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:46.941448 20034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:46.941632 20034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:46.948469 20034 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:46869
I20260928 02:50:46.948498 20101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:46869 every 8 connection(s)
I20260928 02:50:46.949486 20034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:46.952365 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:46.953528 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20034
I20260928 02:50:46.953655 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:50:46.954907 20014 external_mini_cluster.cc:1428] Setting key 684d88edb93174664206348321295882
I20260928 02:50:46.958469 20102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45: Bootstrap starting.
I20260928 02:50:46.960984 20102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:46.961716 20102 log.cc:824] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:46.963538 20102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45: No bootstrap required, opened a new log
I20260928 02:50:46.966506 20102 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cba0b02847427e895d45bc7eb94b45" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46869 } }
I20260928 02:50:46.966727 20102 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:46.966784 20102 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cba0b02847427e895d45bc7eb94b45, State: Initialized, Role: FOLLOWER
I20260928 02:50:46.967209 20102 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cba0b02847427e895d45bc7eb94b45" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46869 } }
I20260928 02:50:46.967334 20102 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:46.967401 20102 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:46.967501 20102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:46.968458 20102 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cba0b02847427e895d45bc7eb94b45" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46869 } }
I20260928 02:50:46.968780 20102 leader_election.cc:302] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cba0b02847427e895d45bc7eb94b45; no voters:
I20260928 02:50:46.969072 20102 leader_election.cc:288] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:46.969220 20107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:46.969424 20107 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [term 1 LEADER]: Becoming Leader. State: Replica: e1cba0b02847427e895d45bc7eb94b45, State: Running, Role: LEADER
I20260928 02:50:46.969726 20107 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cba0b02847427e895d45bc7eb94b45" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46869 } }
I20260928 02:50:46.970360 20102 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:50:46 dist-test-slave-5vc0 krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:46.972626 20108 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cba0b02847427e895d45bc7eb94b45. Latest consensus state: current_term: 1 leader_uuid: "e1cba0b02847427e895d45bc7eb94b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cba0b02847427e895d45bc7eb94b45" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46869 } } }
I20260928 02:50:46.972715 20109 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1cba0b02847427e895d45bc7eb94b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cba0b02847427e895d45bc7eb94b45" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46869 } } }
I20260928 02:50:46.972772 20109 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:46.972759 20108 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45 [sys.catalog]: This master's current role is: LEADER
W20260928 02:50:46.975644 20122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:46.975710 20122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:46.975777 20123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:46.976679 20123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:46.979151 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:46.958116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.19.139.190:46869)
0928 02:50:46.958559 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:46.958579 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:46.959655 (+ 1076us) server_negotiation.cc:407] Connection header received
0928 02:50:46.962808 (+ 3153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:46.962824 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:46.963147 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:46.963449 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:46.965689 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:46.967037 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:46.968376 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:46.968880 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:46.969064 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:46.973705 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:46.973712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:46.973729 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:46.973761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:46.975767 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:46.976356 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:46.976362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:46.976367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:46.976518 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:46.976821 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:46.976825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:46.976826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:46.977210 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:46.977400 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:46.977939 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 02:50:46.978202 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260928 02:50:46.982263 20123 catalog_manager.cc:1347] Generated new cluster ID: 64a495319cf84c17abc6a9fe23e0c69b
I20260928 02:50:46.982331 20123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:47.011139 20123 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:47.012292 20123 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:47.021194 20123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1cba0b02847427e895d45bc7eb94b45: Generated new TSK 0
I20260928 02:50:47.021924 20123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:50:47.057643 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46869
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:47.170147 20128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:47.170425 20128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:47.170495 20128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:47.174696 20128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:47.174775 20128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:47.174862 20128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:50:47.180308 20128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:47.181416 20128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:47.182482 20128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:47.192498 20134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:47.192498 20136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:47.192591 20128 server_base.cc:1034] running on GCE node
W20260928 02:50:47.192490 20133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:47.193200 20128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:47.193902 20128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:47.195091 20128 hybrid_clock.cc:648] HybridClock initialized: now 1790563847195072 us; error 36 us; skew 500 ppm
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:47.198166 20128 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:50:47.199137 20128 webserver.cc:460] Webserver started at http://127.19.139.129:36699/ using document root <none> and password file <none>
I20260928 02:50:47.199707 20128 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:47.199775 20128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:47.199976 20128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:47.202307 20128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "a0aa9b64ff9a431793f98e44faf971e8"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "7eff8607e416e7467a9cf651b3d625af"
server_key_iv: "864bb2cf63b071decb7471a976e54fd9"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.202956 20128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "a0aa9b64ff9a431793f98e44faf971e8"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "7eff8607e416e7467a9cf651b3d625af"
server_key_iv: "864bb2cf63b071decb7471a976e54fd9"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.207063 20128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:50:47.209673 20143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.210960 20128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:47.211063 20128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "a0aa9b64ff9a431793f98e44faf971e8"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "7eff8607e416e7467a9cf651b3d625af"
server_key_iv: "864bb2cf63b071decb7471a976e54fd9"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.211165 20128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:47.225770 20128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:47.228993 20128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:47.229164 20128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:47.229811 20128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:47.230657 20128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:47.230708 20128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.230748 20128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:47.230767 20128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.239257 20128 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:42227
I20260928 02:50:47.239353 20256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:42227 every 8 connection(s)
I20260928 02:50:47.240229 20128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:47.244232 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20128
I20260928 02:50:47.244328 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:47.244565 20014 external_mini_cluster.cc:1428] Setting key 54d5ac2dce3ccd6c50b6dc7b99fc0f85
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:50:47.254001 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.242078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:39051 (local address 127.19.139.190:46869)
0928 02:50:47.242205 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:47.242208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:47.243089 (+ 881us) server_negotiation.cc:407] Connection header received
0928 02:50:47.244457 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:47.244461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:47.244521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:47.244597 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:47.245961 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.246489 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.247650 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.247896 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.247991 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.250636 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:47.250640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:47.250642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:47.250665 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:47.251970 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.252485 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.252489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.252491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.252566 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.252883 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.252887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.252889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.253089 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:47.253182 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:47.253692 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 02:50:47.253828 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:47.254792 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.242463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46869 (local address 127.19.139.129:39051)
0928 02:50:47.242910 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:47.242965 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:47.244184 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:47.244787 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:47.244795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:47.245171 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:47.245822 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:47.245843 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.246604 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.246607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:47.247514 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:47.247523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.248041 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.248045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:47.248275 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.248839 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:47.248855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:47.250475 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:47.252075 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:47.252082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:47.252098 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:47.252395 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:47.252646 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:47.252649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:47.252651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:47.252792 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:47.253185 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:47.253192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:47.253456 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 02:50:47.253621 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:50:47.253969 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:50:47.255986 20257 heartbeater.cc:347] Connected to a master server at 127.19.139.190:46869
I20260928 02:50:47.256269 20257 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:47.256822 20257 heartbeater.cc:511] Master 127.19.139.190:46869 requested a full tablet report, sending...
I20260928 02:50:47.258381 20066 ts_manager.cc:194] Registered new tserver with Master: a0aa9b64ff9a431793f98e44faf971e8 (127.19.139.129:42227)
I20260928 02:50:47.259490 20066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:39051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:50:47.271255 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46869
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:47.380822 20262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:47.381107 20262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:47.381152 20262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:47.385206 20262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:47.385278 20262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:47.385342 20262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:50:47.390756 20262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:47.391782 20262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:47.392815 20262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:47.402706 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:47.402706 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:47.402882 20262 server_base.cc:1034] running on GCE node
W20260928 02:50:47.403132 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:47.403527 20262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:47.404181 20262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:47.405341 20262 hybrid_clock.cc:648] HybridClock initialized: now 1790563847405315 us; error 40 us; skew 500 ppm
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:47.412067 20262 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:50:47.413038 20262 webserver.cc:460] Webserver started at http://127.19.139.130:36059/ using document root <none> and password file <none>
I20260928 02:50:47.413602 20262 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:47.413664 20262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:47.413880 20262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:47.416059 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "972707ba561c4b199eb7217fab6d95d9"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "41f66972aabf20575833436a23023444"
server_key_iv: "3461a60a305293e7f8a571439089b769"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.416795 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "972707ba561c4b199eb7217fab6d95d9"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "41f66972aabf20575833436a23023444"
server_key_iv: "3461a60a305293e7f8a571439089b769"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.420758 20262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:50:47.423244 20277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.424563 20262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:47.424670 20262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "972707ba561c4b199eb7217fab6d95d9"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "41f66972aabf20575833436a23023444"
server_key_iv: "3461a60a305293e7f8a571439089b769"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.424772 20262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:47.439651 20262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:47.442814 20262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:47.443004 20262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:47.443605 20262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:47.444465 20262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:47.444511 20262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.444564 20262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:47.444592 20262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.453733 20262 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:33735
I20260928 02:50:47.453769 20390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:33735 every 8 connection(s)
I20260928 02:50:47.454672 20262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:47.457651 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20262
I20260928 02:50:47.457777 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:47.458065 20014 external_mini_cluster.cc:1428] Setting key 6bdc435880950a7d7219694009281e6e
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:50:47.470386 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.456571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:37881 (local address 127.19.139.190:46869)
0928 02:50:47.456725 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:47.456730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:47.457782 (+ 1052us) server_negotiation.cc:407] Connection header received
0928 02:50:47.459387 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:47.459391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:47.459450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:47.459528 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:47.461420 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.462227 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.463522 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.463851 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.463974 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.466707 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:47.466710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:47.466712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:47.466738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:47.468092 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.468673 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.468675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.468677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.468759 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.469173 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.469177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.469178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.469384 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:47.469504 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:47.470062 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 02:50:47.470217 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:50:47.471472 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.457030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46869 (local address 127.19.139.130:37881)
0928 02:50:47.457564 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:47.457656 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:47.459168 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:47.459837 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:47.459849 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:47.460482 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:47.461226 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:47.461247 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.462407 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.462411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:47.463389 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:47.463399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.464035 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.464039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:47.464264 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.464864 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:47.464881 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:47.466500 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:47.468238 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:47.468245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:47.468259 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:47.468568 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:47.468890 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:47.468893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:47.468894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:47.469057 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:47.469521 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:47.469528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:47.469792 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 02:50:47.470006 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:50:47.470431 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260928 02:50:47.472811 20391 heartbeater.cc:347] Connected to a master server at 127.19.139.190:46869
I20260928 02:50:47.473090 20391 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:47.473804 20391 heartbeater.cc:511] Master 127.19.139.190:46869 requested a full tablet report, sending...
I20260928 02:50:47.474932 20066 ts_manager.cc:194] Registered new tserver with Master: 972707ba561c4b199eb7217fab6d95d9 (127.19.139.130:33735)
I20260928 02:50:47.475690 20066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:37881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:50:47.488360 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46869
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:47.600589 20396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:47.600850 20396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:47.600896 20396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:47.604921 20396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:47.604977 20396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:47.605049 20396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:50:47.610337 20396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:47.611362 20396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:47.612416 20396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:47.621815 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:47.621831 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:47.621918 20396 server_base.cc:1034] running on GCE node
W20260928 02:50:47.621837 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:47.622481 20396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:47.623109 20396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:47.624223 20396 hybrid_clock.cc:648] HybridClock initialized: now 1790563847624194 us; error 41 us; skew 500 ppm
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:47.626816 20396 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:50:47.627696 20396 webserver.cc:460] Webserver started at http://127.19.139.131:43007/ using document root <none> and password file <none>
I20260928 02:50:47.628274 20396 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:47.628340 20396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:47.628523 20396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:47.630609 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "1379a013523949cba30ff31ef539c4b0"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "b3e3e2b8a233a3337a7cb568ebf95dfa"
server_key_iv: "984f00dedfd483f47e8d1e479ec3ad83"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.631161 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "1379a013523949cba30ff31ef539c4b0"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "b3e3e2b8a233a3337a7cb568ebf95dfa"
server_key_iv: "984f00dedfd483f47e8d1e479ec3ad83"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.635345 20396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:50:47.637920 20411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.639181 20396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:47.639305 20396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "1379a013523949cba30ff31ef539c4b0"
format_stamp: "Formatted at 2026-09-28 02:50:47 on dist-test-slave-5vc0"
server_key: "b3e3e2b8a233a3337a7cb568ebf95dfa"
server_key_iv: "984f00dedfd483f47e8d1e479ec3ad83"
server_key_version: "encryptionkey@0"
I20260928 02:50:47.639410 20396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:47.662842 20396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:47.666054 20396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:47.666235 20396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:47.666844 20396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:47.667676 20396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:47.667723 20396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.667775 20396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:47.667801 20396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:47.676190 20396 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:40031
I20260928 02:50:47.676219 20524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:40031 every 8 connection(s)
I20260928 02:50:47.677065 20396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:47.684542 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20396
I20260928 02:50:47.684667 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:47.684912 20014 external_mini_cluster.cc:1428] Setting key 99c9c8928819891950569f42c1d377d0
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:47.690369 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.678830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:60447 (local address 127.19.139.190:46869)
0928 02:50:47.678965 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:47.678968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:47.679803 (+ 835us) server_negotiation.cc:407] Connection header received
0928 02:50:47.681158 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:47.681161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:47.681219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:47.681289 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:47.682469 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.682990 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.684105 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.684417 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.684524 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.687187 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:47.687192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:47.687194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:47.687223 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:47.688620 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.689100 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.689103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.689104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.689177 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.689442 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.689444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.689445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.689616 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:47.689685 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:47.690147 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 02:50:47.690244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:50:47.691174 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.679189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46869 (local address 127.19.139.131:60447)
0928 02:50:47.679658 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:47.679699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:47.680942 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:47.681483 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:47.681490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:47.681846 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:47.682320 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:47.682334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.683102 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.683105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:47.683986 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:47.683994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.684574 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.684577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:47.684774 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.685313 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:47.685336 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:47.687022 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:47.688722 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:47.688727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:47.688739 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:47.689019 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:47.689245 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:47.689247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:47.689248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:47.689378 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:47.689688 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:47.689692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:47.689929 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 02:50:47.690105 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 02:50:47.690364 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":161,"threads_started":1}
I20260928 02:50:47.692232 20525 heartbeater.cc:347] Connected to a master server at 127.19.139.190:46869
I20260928 02:50:47.692485 20525 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:47.692950 20525 heartbeater.cc:511] Master 127.19.139.190:46869 requested a full tablet report, sending...
I20260928 02:50:47.693902 20066 ts_manager.cc:194] Registered new tserver with Master: 1379a013523949cba30ff31ef539c4b0 (127.19.139.131:40031)
I20260928 02:50:47.694494 20066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:60447
I20260928 02:50:47.699308 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:50:47.724331 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.715228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.19.139.129:42227)
0928 02:50:47.715660 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:47.715673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:47.715697 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:50:47.715746 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:47.715752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:47.715957 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:47.716114 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:47.716830 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.717614 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.718430 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.718806 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.718901 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.721146 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:47.721155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:47.721168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:47.721192 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:47.722640 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.723040 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.723046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.723048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.723148 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.723373 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.723376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.723377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.723667 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:47.723829 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:47.724014 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:50:47.724134 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":100,"threads_started":1}
W20260928 02:50:47.725301 20171 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38902
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:50:47.749773 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:47.741730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.19.139.129:42227)
0928 02:50:47.741875 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:47.741880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:47.742067 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:50:47.742173 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:47.742175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:47.742217 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:47.742292 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:47.743149 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.743691 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.744673 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:47.744907 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:47.745005 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:47.747245 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:47.747248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:47.747251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:47.747276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:47.748458 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.748832 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.748834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.748836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.748904 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:47.749095 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:47.749097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:47.749098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:47.749249 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:47.749338 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:47.749550 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:50:47.749648 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:50:47.750727 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20128
I20260928 02:50:47.755314 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20262
I20260928 02:50:47.759495 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20396
I20260928 02:50:47.763733 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20034
2026-09-28T02:50:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2074 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20539](info): set up 2 sockets
Sep 28 02:50:47 dist-test-slave-5vc0 krb5kdc[20539](info): commencing operation
krb5kdc: starting...
W20260928 02:50:48.727677 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:50:48 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:50:48.842641 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45043
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:48.951423 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:48.951687 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:48.951740 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:48.955689 20552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:48.955750 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:48.955776 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:48.955797 20552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:48.955816 20552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:48.960958 20552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:48.961993 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:48.962950 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:48.970824 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:48.970824 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:48.970837 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:48.971024 20552 server_base.cc:1034] running on GCE node
I20260928 02:50:48.971697 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:48.972976 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:48.974131 20552 hybrid_clock.cc:648] HybridClock initialized: now 1790563848974115 us; error 27 us; skew 500 ppm
Sep 28 02:50:48 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:48.976611 20552 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:50:48.977427 20552 webserver.cc:460] Webserver started at http://127.19.139.190:33219/ using document root <none> and password file <none>
I20260928 02:50:48.977959 20552 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:48.978024 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:48.978214 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:48.980136 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "cd5fcd2438b940c298df603820790c21"
format_stamp: "Formatted at 2026-09-28 02:50:48 on dist-test-slave-5vc0"
server_key: "95dd21bc11a2886db0dc639d58fa87b3"
server_key_iv: "933d1f109aeb41dad424c0435cf1240e"
server_key_version: "encryptionkey@0"
I20260928 02:50:48.980698 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "cd5fcd2438b940c298df603820790c21"
format_stamp: "Formatted at 2026-09-28 02:50:48 on dist-test-slave-5vc0"
server_key: "95dd21bc11a2886db0dc639d58fa87b3"
server_key_iv: "933d1f109aeb41dad424c0435cf1240e"
server_key_version: "encryptionkey@0"
I20260928 02:50:48.984087 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:48.986814 20567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:48.987969 20552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:48.988081 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "cd5fcd2438b940c298df603820790c21"
format_stamp: "Formatted at 2026-09-28 02:50:48 on dist-test-slave-5vc0"
server_key: "95dd21bc11a2886db0dc639d58fa87b3"
server_key_iv: "933d1f109aeb41dad424c0435cf1240e"
server_key_version: "encryptionkey@0"
I20260928 02:50:48.988238 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:49.032022 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:49.035153 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:49.035349 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:49.041971 20552 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:45043
I20260928 02:50:49.042002 20619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:45043 every 8 connection(s)
I20260928 02:50:49.042909 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:49.045719 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:49.049082 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20552
I20260928 02:50:49.049181 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:50:49.049399 20014 external_mini_cluster.cc:1428] Setting key bff70b963b88a2479af649b772d0ad99
I20260928 02:50:49.051617 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21: Bootstrap starting.
I20260928 02:50:49.054356 20620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:49.055222 20620 log.cc:824] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:49.057518 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21: No bootstrap required, opened a new log
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:49.061632 20620 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fcd2438b940c298df603820790c21" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45043 } }
I20260928 02:50:49.061893 20620 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:49.061964 20620 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5fcd2438b940c298df603820790c21, State: Initialized, Role: FOLLOWER
I20260928 02:50:49.062438 20620 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fcd2438b940c298df603820790c21" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45043 } }
I20260928 02:50:49.062583 20620 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:49.062685 20620 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:49.062788 20620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:49.063764 20620 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fcd2438b940c298df603820790c21" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45043 } }
I20260928 02:50:49.064121 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.050763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.19.139.190:45043)
0928 02:50:49.051197 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.051216 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.051259 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:50:49.052085 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.052109 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.052564 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.052940 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:49.053559 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.055022 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.056008 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.056715 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.056870 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.059441 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:49.059446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:49.059460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:49.059486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:49.061146 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.061640 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.061644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.061649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.061768 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.062077 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.062079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.062080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.062492 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:49.062683 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.062971 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.063282 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260928 02:50:49.064114 20620 leader_election.cc:302] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd5fcd2438b940c298df603820790c21; no voters:
I20260928 02:50:49.064535 20620 leader_election.cc:288] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:49.064643 20625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:49.064811 20625 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [term 1 LEADER]: Becoming Leader. State: Replica: cd5fcd2438b940c298df603820790c21, State: Running, Role: LEADER
I20260928 02:50:49.065181 20625 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fcd2438b940c298df603820790c21" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45043 } }
I20260928 02:50:49.065856 20620 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:49.067186 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd5fcd2438b940c298df603820790c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fcd2438b940c298df603820790c21" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45043 } } }
I20260928 02:50:49.067232 20627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5fcd2438b940c298df603820790c21. Latest consensus state: current_term: 1 leader_uuid: "cd5fcd2438b940c298df603820790c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fcd2438b940c298df603820790c21" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45043 } } }
I20260928 02:50:49.067291 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:49.067348 20627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:49.067785 20635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:49.070870 20635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:49.076694 20635 catalog_manager.cc:1347] Generated new cluster ID: 6f3a4ae1838945839fa2165ff2996f4a
I20260928 02:50:49.076756 20635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:49.094426 20635 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:49.095432 20635 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:49.103653 20635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd5fcd2438b940c298df603820790c21: Generated new TSK 0
I20260928 02:50:49.104457 20635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:50:49.131124 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45043
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:49.241019 20646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:49.241358 20646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:49.241410 20646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:49.245452 20646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:49.245522 20646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:49.245595 20646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:50:49.250963 20646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:49.252076 20646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:49.253338 20646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:49.263018 20646 server_base.cc:1034] running on GCE node
W20260928 02:50:49.263051 20654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:49.263053 20651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:49.263167 20652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:49.263653 20646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:49.264300 20646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:49.265463 20646 hybrid_clock.cc:648] HybridClock initialized: now 1790563849265445 us; error 32 us; skew 500 ppm
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:49.268252 20646 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:50:49.269120 20646 webserver.cc:460] Webserver started at http://127.19.139.129:35467/ using document root <none> and password file <none>
I20260928 02:50:49.269649 20646 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:49.269707 20646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:49.269874 20646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:49.271862 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "805dd985482c4c52a99ff47e138ba57a"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "8e1ab1dce5578f97a15986244bd59d46"
server_key_iv: "d6847c5f57d0e23e4f2acd04586dd514"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.272405 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "805dd985482c4c52a99ff47e138ba57a"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "8e1ab1dce5578f97a15986244bd59d46"
server_key_iv: "d6847c5f57d0e23e4f2acd04586dd514"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.275894 20646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:50:49.278205 20661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.279347 20646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:49.279446 20646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "805dd985482c4c52a99ff47e138ba57a"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "8e1ab1dce5578f97a15986244bd59d46"
server_key_iv: "d6847c5f57d0e23e4f2acd04586dd514"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.279526 20646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:49.306300 20646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:49.309368 20646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:49.309540 20646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:49.310125 20646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:49.310925 20646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:49.310969 20646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.311015 20646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:49.311045 20646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.319240 20646 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:38821
I20260928 02:50:49.319295 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:38821 every 8 connection(s)
I20260928 02:50:49.320125 20646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:49.327240 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20646
I20260928 02:50:49.327333 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:49.327553 20014 external_mini_cluster.cc:1428] Setting key a4309bf6cf7da5bd8b73ac0e61ffb76c
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:49.333099 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.321927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:47379 (local address 127.19.139.190:45043)
0928 02:50:49.322068 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.322073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.322919 (+ 846us) server_negotiation.cc:407] Connection header received
0928 02:50:49.323772 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.323775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.323821 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.323890 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:49.325078 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.325631 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.326760 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.326992 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.327095 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.329801 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:49.329804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:49.329806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:49.329831 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:49.331119 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.331611 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.331615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.331617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.331690 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.331996 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.332000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.332001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.332205 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:49.332289 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.332816 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.332939 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:50:49.333925 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.322332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45043 (local address 127.19.139.129:47379)
0928 02:50:49.322766 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.322808 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.323587 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.324046 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.324053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.324464 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:49.324943 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.324957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.325755 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.325758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.326643 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.326653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.327144 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.327147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.327300 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.327911 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:49.327927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:49.329642 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:49.331224 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:49.331231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:49.331247 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:49.331519 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:49.331769 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:49.331772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:49.331774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:49.331907 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:49.332296 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:49.332303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:49.332595 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.332777 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.333112 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260928 02:50:49.335081 20775 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45043
I20260928 02:50:49.335342 20775 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:49.335912 20775 heartbeater.cc:511] Master 127.19.139.190:45043 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:50:49.337457 20584 ts_manager.cc:194] Registered new tserver with Master: 805dd985482c4c52a99ff47e138ba57a (127.19.139.129:38821)
I20260928 02:50:49.338589 20584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:47379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:50:49.355082 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45043
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:49.462113 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:49.462385 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:49.462438 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:49.466645 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:49.466718 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:49.466791 20780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:50:49.472308 20780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:49.473384 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:49.474407 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:49.483870 20780 server_base.cc:1034] running on GCE node
W20260928 02:50:49.483770 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:49.483770 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:49.483790 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:49.484509 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:49.485137 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:49.486297 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790563849486268 us; error 38 us; skew 500 ppm
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:49.488885 20780 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:50:49.489696 20780 webserver.cc:460] Webserver started at http://127.19.139.130:36069/ using document root <none> and password file <none>
I20260928 02:50:49.490237 20780 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:49.490294 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:49.490514 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:49.492676 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "d054bbfb982249a688383c10e71a7377"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "b6af35b962abf47d30df0413928c8bc4"
server_key_iv: "8f4b0c37f0bda93ecfbd6e6eaa7738c4"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.493192 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "d054bbfb982249a688383c10e71a7377"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "b6af35b962abf47d30df0413928c8bc4"
server_key_iv: "8f4b0c37f0bda93ecfbd6e6eaa7738c4"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.496668 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:50:49.498840 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.499972 20780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:49.500061 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "d054bbfb982249a688383c10e71a7377"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "b6af35b962abf47d30df0413928c8bc4"
server_key_iv: "8f4b0c37f0bda93ecfbd6e6eaa7738c4"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.500172 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:49.522105 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:49.525290 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:49.525475 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:49.526059 20780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:49.526856 20780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:49.526901 20780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.526945 20780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:49.526974 20780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.534927 20780 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:33815
I20260928 02:50:49.534979 20908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:33815 every 8 connection(s)
I20260928 02:50:49.535784 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:49.541021 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20780
I20260928 02:50:49.541117 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:49.541358 20014 external_mini_cluster.cc:1428] Setting key 9c851f934881de571af52e39b8a6a1ee
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:49.548559 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.537540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:53625 (local address 127.19.139.190:45043)
0928 02:50:49.537677 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.537682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.538528 (+ 846us) server_negotiation.cc:407] Connection header received
0928 02:50:49.539393 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.539397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.539460 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.539527 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:49.540691 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.541287 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.542395 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.542616 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.542701 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.545107 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:49.545111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:49.545114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:49.545141 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:49.546600 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.547068 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.547072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.547074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.547166 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.547449 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.547453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.547454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.547685 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:49.547787 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.548255 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.548383 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:50:49.549316 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.537933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45043 (local address 127.19.139.130:53625)
0928 02:50:49.538386 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.538429 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.539202 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.539697 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.539704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.540058 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:49.540561 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.540574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.541397 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.541400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.542284 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.542293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.542743 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.542747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.542900 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.543430 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:49.543447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:49.544944 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:49.546700 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:49.546704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:49.546717 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:49.546972 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:49.547245 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:49.547246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:49.547248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:49.547369 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:49.547770 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:49.547775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:49.548013 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.548195 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.548459 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:50:49.550423 20909 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45043
I20260928 02:50:49.550659 20909 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:49.551113 20909 heartbeater.cc:511] Master 127.19.139.190:45043 requested a full tablet report, sending...
I20260928 02:50:49.552134 20584 ts_manager.cc:194] Registered new tserver with Master: d054bbfb982249a688383c10e71a7377 (127.19.139.130:33815)
I20260928 02:50:49.552785 20584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:53625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:50:49.567849 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45043
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:49.675253 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:49.675516 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:49.675562 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:49.679874 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:49.679960 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:49.680043 20914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:50:49.685470 20914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45043
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:49.686427 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:49.687338 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:49.696758 20919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:49.696754 20920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:49.696918 20914 server_base.cc:1034] running on GCE node
W20260928 02:50:49.696758 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:49.697480 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:49.698143 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:49.699297 20914 hybrid_clock.cc:648] HybridClock initialized: now 1790563849699269 us; error 40 us; skew 500 ppm
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:49.701988 20914 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:50:49.702901 20914 webserver.cc:460] Webserver started at http://127.19.139.131:35831/ using document root <none> and password file <none>
I20260928 02:50:49.703454 20914 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:49.703521 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:49.703771 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:49.705876 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "cc5dea7fb1fc48e59430919e7aa04180"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "3749c275074280fdb4a7b86edf5ca5d2"
server_key_iv: "f1ce4260e73f0686b88553efd0b9ddbc"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.706439 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "cc5dea7fb1fc48e59430919e7aa04180"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "3749c275074280fdb4a7b86edf5ca5d2"
server_key_iv: "f1ce4260e73f0686b88553efd0b9ddbc"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.710146 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:50:49.712561 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.713784 20914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:49.713879 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "cc5dea7fb1fc48e59430919e7aa04180"
format_stamp: "Formatted at 2026-09-28 02:50:49 on dist-test-slave-5vc0"
server_key: "3749c275074280fdb4a7b86edf5ca5d2"
server_key_iv: "f1ce4260e73f0686b88553efd0b9ddbc"
server_key_version: "encryptionkey@0"
I20260928 02:50:49.713968 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:49.729583 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:49.732903 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:49.733084 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:49.733661 20914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:49.734560 20914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:49.734609 20914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.734649 20914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:49.734663 20914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:49.743191 20914 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:33127
I20260928 02:50:49.743214 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:33127 every 8 connection(s)
I20260928 02:50:49.744076 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:49.753778 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 20914
I20260928 02:50:49.753926 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:49.754192 20014 external_mini_cluster.cc:1428] Setting key 1d63e85f2d68aad79e8d9244f5768ff8
I20260928 02:50:49.757053 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.745944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:42255 (local address 127.19.139.190:45043)
0928 02:50:49.746144 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.746149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.747045 (+ 896us) server_negotiation.cc:407] Connection header received
0928 02:50:49.747893 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.747896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.747952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.748024 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:49.749292 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.749851 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.750944 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.751146 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.751215 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.753523 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:49.753527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:49.753530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:49.753576 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:49.755037 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.755618 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.755621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.755622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.755697 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.756053 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.756056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.756058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.756297 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:49.756381 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.756825 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.756937 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 02:50:49.757886 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.746365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45043 (local address 127.19.139.131:42255)
0928 02:50:49.746860 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.746905 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.747695 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.748214 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.748223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.748604 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:49.749118 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.749132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.749970 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.749972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.750851 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.750859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.751256 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.751258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.751395 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.751922 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:49.751938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:49.753369 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:49.755181 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:49.755186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:49.755198 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:49.755536 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:49.755773 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:49.755776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:49.755777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:49.755972 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:49.756394 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:49.756399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:49.756632 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.756793 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.757058 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260928 02:50:49.758894 21043 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45043
I20260928 02:50:49.759119 21043 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:49.759598 21043 heartbeater.cc:511] Master 127.19.139.190:45043 requested a full tablet report, sending...
I20260928 02:50:49.760555 20584 ts_manager.cc:194] Registered new tserver with Master: cc5dea7fb1fc48e59430919e7aa04180 (127.19.139.131:33127)
I20260928 02:50:49.761096 20584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:42255
I20260928 02:50:49.767928 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:49.777837 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.771240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.19.139.190:45043)
0928 02:50:49.771381 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.771385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.771506 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:50:49.771604 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.771606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.771645 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.771702 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:49.772416 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.772946 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.773721 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.773947 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.774019 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.775189 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:49.775192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:49.775194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:49.775217 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:49.776542 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.776879 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.776881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.776882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.776961 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.777149 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.777151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.777152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.777353 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:49.777456 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.777607 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.777694 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:50:49.782390 20584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:49.785545 20584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:49.799255 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.793442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38821 (local address 127.0.0.1:38422)
0928 02:50:49.793783 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.793833 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.793947 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.794920 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.794926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.794949 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:49.795086 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.795093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.796578 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.796581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.797702 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.797713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.798699 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.798703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.798816 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.798853 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.798967 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.799103 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":49,"thread_start_us":90,"threads_started":1}
I20260928 02:50:49.799266 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.793565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.19.139.129:38821)
0928 02:50:49.794399 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.794414 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.794449 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:49.794529 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.794535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.794757 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.794919 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:49.795213 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.796445 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.797839 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.798584 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.798700 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.798846 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.799012 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.799116 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":87,"threads_started":1}
I20260928 02:50:49.799561 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.793785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:33127 (local address 127.0.0.1:47128)
0928 02:50:49.794048 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.794060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.794138 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.795201 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.795204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.795220 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:49.795306 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.795312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.796757 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.796760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.798064 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.798074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.799163 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.799167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.799262 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.799294 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.799342 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.799424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260928 02:50:49.799858 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.794321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.19.139.130:33815)
0928 02:50:49.795273 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.795284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.795306 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:50:49.795373 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.795378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.795551 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.795677 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:49.795857 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.796687 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.798200 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.799034 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.799151 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.799273 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.799627 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.799727 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":142,"threads_started":1}
I20260928 02:50:49.800102 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.794171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33815 (local address 127.0.0.1:41372)
0928 02:50:49.794400 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.794419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.794504 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.795671 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.795674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.795690 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:49.795773 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.795779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.796799 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.796802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.798065 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.798074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.799326 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.799329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.799407 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.799426 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.799485 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.799542 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":111,"thread_start_us":105,"threads_started":1}
I20260928 02:50:49.800266 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.794469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.19.139.131:33127)
0928 02:50:49.794777 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.794785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.794804 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:50:49.794866 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.794870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.795057 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.795438 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:49.795576 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.796642 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.798200 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.799034 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.799845 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.799972 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.800020 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.800109 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":141,"threads_started":1}
I20260928 02:50:49.802567 20709 tablet_service.cc:1467] Processing CreateTablet for tablet 32aa390118c849febe2216546ea84b9b (DEFAULT_TABLE table=test-table [id=9239f8d1536b45ca93db2bafc8163bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:49.802922 20977 tablet_service.cc:1467] Processing CreateTablet for tablet 32aa390118c849febe2216546ea84b9b (DEFAULT_TABLE table=test-table [id=9239f8d1536b45ca93db2bafc8163bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:49.802927 20843 tablet_service.cc:1467] Processing CreateTablet for tablet 32aa390118c849febe2216546ea84b9b (DEFAULT_TABLE table=test-table [id=9239f8d1536b45ca93db2bafc8163bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:49.803668 20709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32aa390118c849febe2216546ea84b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:49.803952 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32aa390118c849febe2216546ea84b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:49.803954 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32aa390118c849febe2216546ea84b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:49.810665 21060 tablet_bootstrap.cc:492] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a: Bootstrap starting.
I20260928 02:50:49.811579 21061 tablet_bootstrap.cc:492] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: Bootstrap starting.
I20260928 02:50:49.812511 21062 tablet_bootstrap.cc:492] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: Bootstrap starting.
I20260928 02:50:49.812860 21060 tablet_bootstrap.cc:654] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:49.813572 21061 tablet_bootstrap.cc:654] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:49.813890 21060 log.cc:824] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:49.814694 21061 log.cc:824] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:49.814855 21062 tablet_bootstrap.cc:654] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:49.815657 21062 log.cc:824] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:49.815821 21060 tablet_bootstrap.cc:492] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a: No bootstrap required, opened a new log
I20260928 02:50:49.816045 21060 ts_tablet_manager.cc:1397] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:50:49.816946 21061 tablet_bootstrap.cc:492] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: No bootstrap required, opened a new log
I20260928 02:50:49.817121 21061 ts_tablet_manager.cc:1397] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 02:50:49.817638 21062 tablet_bootstrap.cc:492] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: No bootstrap required, opened a new log
I20260928 02:50:49.817812 21062 ts_tablet_manager.cc:1397] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:50:49.819947 21061 raft_consensus.cc:348] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.819967 21060 raft_consensus.cc:348] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.820199 21061 raft_consensus.cc:374] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:49.820237 21060 raft_consensus.cc:374] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:49.820269 21061 raft_consensus.cc:729] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5dea7fb1fc48e59430919e7aa04180, State: Initialized, Role: FOLLOWER
I20260928 02:50:49.820325 21060 raft_consensus.cc:729] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805dd985482c4c52a99ff47e138ba57a, State: Initialized, Role: FOLLOWER
I20260928 02:50:49.820696 21061 consensus_queue.cc:260] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.820745 21060 consensus_queue.cc:260] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.821338 21043 heartbeater.cc:503] Master 127.19.139.190:45043 was elected leader, sending a full tablet report...
I20260928 02:50:49.821676 21061 ts_tablet_manager.cc:1428] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:50:49.821491 21062 raft_consensus.cc:348] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.821755 21062 raft_consensus.cc:374] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:49.821849 21062 raft_consensus.cc:729] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d054bbfb982249a688383c10e71a7377, State: Initialized, Role: FOLLOWER
I20260928 02:50:49.821864 21060 ts_tablet_manager.cc:1428] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:50:49.822183 20775 heartbeater.cc:503] Master 127.19.139.190:45043 was elected leader, sending a full tablet report...
I20260928 02:50:49.822328 21062 consensus_queue.cc:260] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.823247 21062 ts_tablet_manager.cc:1428] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:50:49.823585 20909 heartbeater.cc:503] Master 127.19.139.190:45043 was elected leader, sending a full tablet report...
I20260928 02:50:49.946133 21068 raft_consensus.cc:482] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:49.946350 21068 raft_consensus.cc:504] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.947561 21068 leader_election.cc:288] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 805dd985482c4c52a99ff47e138ba57a (127.19.139.129:38821), cc5dea7fb1fc48e59430919e7aa04180 (127.19.139.131:33127)
I20260928 02:50:49.952302 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.947856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:47263 (local address 127.19.139.129:38821)
0928 02:50:49.947977 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.947982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.948065 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:50:49.948204 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.948208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.948258 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.948342 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:49.948596 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.949365 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.951017 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.951635 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.951756 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.951810 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.952071 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.952143 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:50:49.952301 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.947895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:58991 (local address 127.19.139.131:33127)
0928 02:50:49.948016 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.948020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.948277 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:50:49.948357 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.948360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.948403 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.948476 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:49.948709 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.949440 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.951016 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.951638 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.951756 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.951810 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.952071 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.952146 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:50:49.952339 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.947780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38821 (local address 127.19.139.130:47263)
0928 02:50:49.947975 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.947989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.948083 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.948363 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.948366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.948382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:49.948489 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.948495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.949486 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.949490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.950886 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.950896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.951752 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.951756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.951949 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.951986 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.952048 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.952169 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":160}
I20260928 02:50:49.952386 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.947812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:33127 (local address 127.19.139.130:58991)
0928 02:50:49.948193 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:49.948205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:49.948283 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:49.948492 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:49.948494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:49.948508 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:49.948612 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.948619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.949540 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.949543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.950885 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:49.950896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.951752 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.951756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:49.951934 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.951971 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:50:49.952048 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:50:49.952146 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":63,"thread_start_us":140,"threads_started":1}
I20260928 02:50:49.952997 20729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32aa390118c849febe2216546ea84b9b" candidate_uuid: "d054bbfb982249a688383c10e71a7377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805dd985482c4c52a99ff47e138ba57a" is_pre_election: true
I20260928 02:50:49.953001 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32aa390118c849febe2216546ea84b9b" candidate_uuid: "d054bbfb982249a688383c10e71a7377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5dea7fb1fc48e59430919e7aa04180" is_pre_election: true
I20260928 02:50:49.953328 20997 raft_consensus.cc:2454] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d054bbfb982249a688383c10e71a7377 in term 0.
I20260928 02:50:49.953328 20729 raft_consensus.cc:2454] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d054bbfb982249a688383c10e71a7377 in term 0.
I20260928 02:50:49.953837 20798 leader_election.cc:302] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 805dd985482c4c52a99ff47e138ba57a, cc5dea7fb1fc48e59430919e7aa04180, d054bbfb982249a688383c10e71a7377; no voters:
I20260928 02:50:49.954146 21068 raft_consensus.cc:2781] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:49.954232 21068 raft_consensus.cc:482] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:49.954298 21068 raft_consensus.cc:3037] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:49.955295 21068 raft_consensus.cc:504] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.955677 21068 leader_election.cc:288] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [CANDIDATE]: Term 1 election: Requested vote from peers 805dd985482c4c52a99ff47e138ba57a (127.19.139.129:38821), cc5dea7fb1fc48e59430919e7aa04180 (127.19.139.131:33127)
I20260928 02:50:49.956075 20729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32aa390118c849febe2216546ea84b9b" candidate_uuid: "d054bbfb982249a688383c10e71a7377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805dd985482c4c52a99ff47e138ba57a"
I20260928 02:50:49.956075 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32aa390118c849febe2216546ea84b9b" candidate_uuid: "d054bbfb982249a688383c10e71a7377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5dea7fb1fc48e59430919e7aa04180"
I20260928 02:50:49.956241 20997 raft_consensus.cc:3037] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:49.956241 20729 raft_consensus.cc:3037] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:49.957518 20997 raft_consensus.cc:2454] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d054bbfb982249a688383c10e71a7377 in term 1.
I20260928 02:50:49.957518 20729 raft_consensus.cc:2454] T 32aa390118c849febe2216546ea84b9b P 805dd985482c4c52a99ff47e138ba57a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d054bbfb982249a688383c10e71a7377 in term 1.
I20260928 02:50:49.957855 20796 leader_election.cc:302] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805dd985482c4c52a99ff47e138ba57a, d054bbfb982249a688383c10e71a7377; no voters:
I20260928 02:50:49.958072 21068 raft_consensus.cc:2781] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:49.958283 21068 raft_consensus.cc:686] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [term 1 LEADER]: Becoming Leader. State: Replica: d054bbfb982249a688383c10e71a7377, State: Running, Role: LEADER
I20260928 02:50:49.958616 21068 consensus_queue.cc:237] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } }
I20260928 02:50:49.961928 20584 catalog_manager.cc:5495] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377 reported cstate change: term changed from 0 to 1, leader changed from <none> to d054bbfb982249a688383c10e71a7377 (127.19.139.130). New cstate: current_term: 1 leader_uuid: "d054bbfb982249a688383c10e71a7377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805dd985482c4c52a99ff47e138ba57a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d054bbfb982249a688383c10e71a7377" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5dea7fb1fc48e59430919e7aa04180" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33127 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:50:49.990037 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:49.981924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.19.139.129:38821)
0928 02:50:49.982067 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:49.982072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:49.982177 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:50:49.982286 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:49.982289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:49.982334 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:49.982403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:49.983045 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.983632 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.984485 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:49.984689 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:49.984775 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:49.987081 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:49.987094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:49.987109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:49.987135 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:49.988598 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.988973 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.988977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.988979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.989063 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:49.989241 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:49.989243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:49.989244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:49.989506 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:49.989658 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:49.989781 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:50:49.989876 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:50:49.995011 21044 tablet.cc:2369] T 32aa390118c849febe2216546ea84b9b P cc5dea7fb1fc48e59430919e7aa04180: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:50:49 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:50:50 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:50:50.015267 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:50.007370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.19.139.129:38821)
0928 02:50:50.007512 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:50.007518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:50.007651 (+ 133us) server_negotiation.cc:407] Connection header received
0928 02:50:50.007784 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:50.007786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:50.007827 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:50.007892 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:50.008627 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:50.009131 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:50.009933 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:50.010151 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:50.010234 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:50.012537 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:50.012540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:50.012542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:50.012568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:50.013725 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:50.014095 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:50.014098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:50.014099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:50.014176 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:50.014407 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:50.014411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:50.014412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:50.014580 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:50.014694 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:50.014981 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:50:50.015118 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:50:50 dist-test-slave-5vc0 krb5kdc[20539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:50:50 dist-test-slave-5vc0 krb5kdc[20539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
W20260928 02:50:50.036895 20910 tablet.cc:2369] T 32aa390118c849febe2216546ea84b9b P d054bbfb982249a688383c10e71a7377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:50.039063 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:50.031454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.19.139.129:38821)
0928 02:50:50.031614 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:50.031618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:50.031684 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:50:50.031796 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:50.031799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:50.031841 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:50.031909 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:50.032621 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:50.033114 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:50.033945 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:50.034137 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:50.034203 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:50.036407 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:50.036410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:50.036412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:50.036464 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:50.037738 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:50.038117 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:50.038120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:50.038121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:50.038185 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:50.038404 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:50.038406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:50.038407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:50.038573 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:50.038665 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:50.038828 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:50:50.038917 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:50:50.040454 20689 tablet_service.cc:2908] Rejecting scan request for tablet 32aa390118c849febe2216546ea84b9b: Uninitialized: clean time has not yet been initialized
I20260928 02:50:50.042577 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20646
I20260928 02:50:50.049057 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20780
I20260928 02:50:50.053756 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20914
I20260928 02:50:50.058648 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 20552
2026-09-28T02:50:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2295 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:50 dist-test-slave-5vc0 krb5kdc[21084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:50 dist-test-slave-5vc0 krb5kdc[21084](info): set up 2 sockets
Sep 28 02:50:50 dist-test-slave-5vc0 krb5kdc[21084](info): commencing operation
krb5kdc: starting...
W20260928 02:50:51.019660 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:50:51.139611 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:39019
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:51.251037 21097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:51.251325 21097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:51.251364 21097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:51.255475 21097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:51.255537 21097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:50:51.255553 21097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:51.255570 21097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:51.255585 21097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:51.255607 21097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:51.261063 21097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:39019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:51.262152 21097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:51.263180 21097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:51.271778 21103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:51.271754 21105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.271833 21097 server_base.cc:1034] running on GCE node
W20260928 02:50:51.271763 21102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.272614 21097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:51.273757 21097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:51.274912 21097 hybrid_clock.cc:648] HybridClock initialized: now 1790563851274886 us; error 39 us; skew 500 ppm
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:51.277603 21097 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:50:51.278518 21097 webserver.cc:460] Webserver started at http://127.19.139.190:42167/ using document root <none> and password file <none>
I20260928 02:50:51.279069 21097 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:51.279134 21097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:51.279316 21097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:51.281405 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "5ae2f68ac6834e3790ec44811074cdbf"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "f5f222b447aed4e28ffbe5f0f3065b03"
server_key_iv: "2c4113dae90eb610a6d96f9a30aec7b7"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.281956 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "5ae2f68ac6834e3790ec44811074cdbf"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "f5f222b447aed4e28ffbe5f0f3065b03"
server_key_iv: "2c4113dae90eb610a6d96f9a30aec7b7"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.285732 21097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:51.288280 21112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.289523 21097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:51.289644 21097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "5ae2f68ac6834e3790ec44811074cdbf"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "f5f222b447aed4e28ffbe5f0f3065b03"
server_key_iv: "2c4113dae90eb610a6d96f9a30aec7b7"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.289744 21097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:51.302850 21097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:51.306311 21097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:51.306499 21097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:51.314253 21097 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:39019
I20260928 02:50:51.314260 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:39019 every 8 connection(s)
I20260928 02:50:51.315260 21097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:51.315824 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21097
I20260928 02:50:51.315938 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:50:51.316233 20014 external_mini_cluster.cc:1428] Setting key dfd8089e6d84fec8a5d1cfdad92c7129
I20260928 02:50:51.318646 21175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:51.325572 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: Bootstrap starting.
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:51.328408 21175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:51.329720 21175 log.cc:824] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:51.331460 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.317674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.19.139.190:39019)
0928 02:50:51.318144 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:51.318164 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:51.318209 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:50:51.319051 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:51.319076 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:51.319522 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:51.319905 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:51.320667 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.322199 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.323098 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.323771 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.323953 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.326552 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:51.326559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:51.326577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:51.326612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:51.328540 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.328985 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.328992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.328999 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.329154 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.329403 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.329405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.329407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.329957 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:51.330147 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:51.330378 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:50:51.330674 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260928 02:50:51.331903 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: No bootstrap required, opened a new log
I20260928 02:50:51.334888 21175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae2f68ac6834e3790ec44811074cdbf" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 39019 } }
I20260928 02:50:51.335116 21175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:51.335181 21175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae2f68ac6834e3790ec44811074cdbf, State: Initialized, Role: FOLLOWER
I20260928 02:50:51.335610 21175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae2f68ac6834e3790ec44811074cdbf" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 39019 } }
I20260928 02:50:51.335749 21175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:51.335813 21175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:51.335909 21175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:51.336879 21175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae2f68ac6834e3790ec44811074cdbf" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 39019 } }
I20260928 02:50:51.337200 21175 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae2f68ac6834e3790ec44811074cdbf; no voters:
I20260928 02:50:51.337493 21175 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:51.337635 21180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:51.337839 21180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [term 1 LEADER]: Becoming Leader. State: Replica: 5ae2f68ac6834e3790ec44811074cdbf, State: Running, Role: LEADER
I20260928 02:50:51.338178 21180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae2f68ac6834e3790ec44811074cdbf" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 39019 } }
I20260928 02:50:51.338647 21175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:51.339715 21181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae2f68ac6834e3790ec44811074cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae2f68ac6834e3790ec44811074cdbf" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 39019 } } }
I20260928 02:50:51.339764 21182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae2f68ac6834e3790ec44811074cdbf. Latest consensus state: current_term: 1 leader_uuid: "5ae2f68ac6834e3790ec44811074cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae2f68ac6834e3790ec44811074cdbf" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 39019 } } }
I20260928 02:50:51.339835 21181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:51.339862 21182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:51.340224 21189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:51.343418 21189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:51.349171 21189 catalog_manager.cc:1347] Generated new cluster ID: 2f359901626f499f8d6e32b9d75a5170
I20260928 02:50:51.349252 21189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:51.356709 21189 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:51.357643 21189 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:51.366045 21189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: Generated new TSK 0
I20260928 02:50:51.366758 21189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:50:51.399894 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:39019
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:51.512903 21201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:51.513188 21201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:51.513233 21201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:51.517345 21201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:51.517412 21201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:51.517452 21201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:50:51.517490 21201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:50:51.522857 21201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:51.524031 21201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:51.525143 21201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:51.534804 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:51.534796 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:51.534796 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.535198 21201 server_base.cc:1034] running on GCE node
I20260928 02:50:51.535602 21201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:51.536311 21201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:51.537482 21201 hybrid_clock.cc:648] HybridClock initialized: now 1790563851537462 us; error 31 us; skew 500 ppm
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:51.540437 21201 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:50:51.541340 21201 webserver.cc:460] Webserver started at http://127.19.139.129:33281/ using document root <none> and password file <none>
I20260928 02:50:51.541886 21201 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:51.541949 21201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:51.542145 21201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:51.544303 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "b6fa5e12a298432188fab4d7287c0de5"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "718c525dab16079b601faa45aed3ed1f"
server_key_iv: "e442ef89b765bca8010ea9a54aaf5452"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.544907 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "b6fa5e12a298432188fab4d7287c0de5"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "718c525dab16079b601faa45aed3ed1f"
server_key_iv: "e442ef89b765bca8010ea9a54aaf5452"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.549005 21201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:50:51.551637 21216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.553033 21201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:50:51.553164 21201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "b6fa5e12a298432188fab4d7287c0de5"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "718c525dab16079b601faa45aed3ed1f"
server_key_iv: "e442ef89b765bca8010ea9a54aaf5452"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.553282 21201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:51.580688 21201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:51.584036 21201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:51.584247 21201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:51.585701 21201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:51.585752 21201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.585806 21201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:51.585834 21201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.595626 21201 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:44093
I20260928 02:50:51.595733 21331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:44093 every 8 connection(s)
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:51.596683 21201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:51.596899 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21201
I20260928 02:50:51.597023 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:51.597303 20014 external_mini_cluster.cc:1428] Setting key 5ba67877813c2db14a35806f84f9c735
I20260928 02:50:51.600575 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.586594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:42557 (local address 127.19.139.190:39019)
0928 02:50:51.586733 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:51.586737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:51.587655 (+ 918us) server_negotiation.cc:407] Connection header received
0928 02:50:51.588895 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:51.588899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:51.588967 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:51.589049 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:51.590702 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.591518 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.593263 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.593532 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.593637 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.596707 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:51.596711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:51.596714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:51.596748 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:51.598356 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.598960 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.598964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.598966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.599067 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.599388 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.599391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.599393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.599667 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:51.599759 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:51.600303 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 02:50:51.600454 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:50:51.601699 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.587031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.19.139.129:42557)
0928 02:50:51.587499 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:51.587550 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:51.588670 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:51.589281 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:51.589293 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:51.589790 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:51.590547 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.590571 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.591682 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.591686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.593144 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.593156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.593705 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.593711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.593946 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.594661 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:51.594681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:51.596494 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:51.598468 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.598473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.598489 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.598847 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.599153 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.599155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.599157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.599293 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.599766 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:51.599772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:51.600037 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:50:51.600241 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 02:50:51.600621 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260928 02:50:51.602865 21332 heartbeater.cc:347] Connected to a master server at 127.19.139.190:39019
I20260928 02:50:51.603011 21332 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:51.603554 21332 heartbeater.cc:511] Master 127.19.139.190:39019 requested a full tablet report, sending...
I20260928 02:50:51.605036 21128 ts_manager.cc:194] Registered new tserver with Master: b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093)
I20260928 02:50:51.606235 21128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:42557
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:50:51.613330 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.605333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.19.139.129:36333)
0928 02:50:51.606944 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:51.606958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:51.607063 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:51.607343 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:51.607346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:51.607529 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:51.607628 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.607633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.608425 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.608428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.609023 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.609032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.609455 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.609458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.609543 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.609967 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:51.609977 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:51.610282 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:51.611880 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.611883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.611885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.612087 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.612384 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.612386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.612388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.612465 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.612814 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:51.612818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:51.612899 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 02:50:51.612992 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:50:51.613136 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":115,"threads_started":1}
I20260928 02:50:51.613324 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.605406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:36333 (local address 127.19.139.190:39019)
0928 02:50:51.605532 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:51.605536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:51.607047 (+ 1511us) server_negotiation.cc:407] Connection header received
0928 02:50:51.607202 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:51.607205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:51.607250 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:51.607312 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:51.607733 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.608307 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.609127 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.609354 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.609454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.610415 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:51.610418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:51.610421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:51.610444 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:51.611774 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.612210 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.612214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.612216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.612298 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.612546 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.612550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.612551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.612725 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:51.612811 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:51.613061 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:50:51.613171 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:50:51.627362 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:39019
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:51.736656 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:51.736929 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:51.736992 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:51.741101 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:51.741174 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:51.741230 21341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:50:51.741286 21341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:50:51.746728 21341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:51.747822 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:51.748956 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:51.758533 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:51.758579 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.758608 21341 server_base.cc:1034] running on GCE node
W20260928 02:50:51.758555 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.759140 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:51.759763 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:51.760913 21341 hybrid_clock.cc:648] HybridClock initialized: now 1790563851760898 us; error 38 us; skew 500 ppm
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:51.763463 21341 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:50:51.764326 21341 webserver.cc:460] Webserver started at http://127.19.139.130:42537/ using document root <none> and password file <none>
I20260928 02:50:51.764853 21341 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:51.764917 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:51.765097 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:51.767072 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "b392be2606374e418766369fb68e560f"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "9c4508f7d872c725015f4c493e9b7c4d"
server_key_iv: "a5d1e3aa4ec39704b63f94b7600fc553"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.767592 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "b392be2606374e418766369fb68e560f"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "9c4508f7d872c725015f4c493e9b7c4d"
server_key_iv: "a5d1e3aa4ec39704b63f94b7600fc553"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.771138 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:50:51.773543 21356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.774775 21341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:51.774883 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "b392be2606374e418766369fb68e560f"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "9c4508f7d872c725015f4c493e9b7c4d"
server_key_iv: "a5d1e3aa4ec39704b63f94b7600fc553"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.774983 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:51.796717 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:51.799605 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:51.799798 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:51.801326 21341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:51.801383 21341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.801549 21341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:51.801584 21341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:51.812137 21341 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:39273
I20260928 02:50:51.812191 21471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:39273 every 8 connection(s)
I20260928 02:50:51.813217 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:51.813524 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21341
I20260928 02:50:51.813639 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:51.813913 20014 external_mini_cluster.cc:1428] Setting key b66f22ddf258ed0f2b75666314b15667
I20260928 02:50:51.816232 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.802307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:42261 (local address 127.19.139.190:39019)
0928 02:50:51.802428 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:51.802434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:51.803560 (+ 1126us) server_negotiation.cc:407] Connection header received
0928 02:50:51.804747 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:51.804751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:51.804813 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:51.804897 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:51.806634 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.807404 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.808699 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.809033 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.809139 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.812129 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:51.812133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:51.812135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:51.812198 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:51.813902 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.814487 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.814491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.814493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.814590 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.815006 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.815009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.815011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.815250 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:51.815340 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:51.815885 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:50:51.816044 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:50:51.817063 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.802903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.19.139.130:42261)
0928 02:50:51.803401 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:51.803454 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:51.804472 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:51.805123 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:51.805134 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:51.805635 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:51.806395 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.806417 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.807528 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.807532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.808543 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.808554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.809189 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.809194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.809408 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.810060 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:51.810083 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:51.811925 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:51.814020 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.814027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.814042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.814394 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.814690 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.814693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.814695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.814887 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.815357 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:51.815365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:51.815627 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 02:50:51.815805 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 02:50:51.816096 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260928 02:50:51.818344 21472 heartbeater.cc:347] Connected to a master server at 127.19.139.190:39019
I20260928 02:50:51.818504 21472 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:51.818993 21472 heartbeater.cc:511] Master 127.19.139.190:39019 requested a full tablet report, sending...
I20260928 02:50:51.820180 21129 ts_manager.cc:194] Registered new tserver with Master: b392be2606374e418766369fb68e560f (127.19.139.130:39273)
I20260928 02:50:51.820961 21129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:42261
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:50:51.828513 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.820819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:52177 (local address 127.19.139.190:39019)
0928 02:50:51.820936 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:51.820940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:51.821184 (+ 244us) server_negotiation.cc:407] Connection header received
0928 02:50:51.821336 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:51.821339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:51.821387 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:51.821452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:51.822060 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.822859 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.824034 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.824372 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.824458 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.825508 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:51.825511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:51.825513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:51.825536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:51.826850 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.827325 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.827327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.827328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.827403 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:51.827695 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:51.827697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:51.827698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:51.827925 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:51.828000 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:51.828284 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:50:51.828390 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:50:51.828594 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:51.820746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.19.139.130:52177)
0928 02:50:51.821094 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:51.821109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:51.821217 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:51.821588 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:51.821591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:51.821810 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:51.821932 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.821940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.823017 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.823020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.823906 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:51.823917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:51.824485 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:51.824488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:51.824558 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:51.825005 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:51.825016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:51.825368 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:51.826964 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.826968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.826970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.827232 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.827494 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:51.827497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:51.827499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:51.827615 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:51.828011 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:51.828014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:51.828114 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:50:51.828245 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 02:50:51.828404 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:50:51.844076 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:39019
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:51.954646 21481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:51.955054 21481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:51.955119 21481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:51.959406 21481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:51.959483 21481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:51.959524 21481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:50:51.959560 21481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:50:51.964977 21481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:51.965986 21481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:51.966956 21481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:51.976207 21486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:51.976199 21487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.976264 21481 server_base.cc:1034] running on GCE node
W20260928 02:50:51.976199 21489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:51.976853 21481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:51.977514 21481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:51.978685 21481 hybrid_clock.cc:648] HybridClock initialized: now 1790563851978663 us; error 34 us; skew 500 ppm
Sep 28 02:50:51 dist-test-slave-5vc0 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:51.981673 21481 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:50:51.982616 21481 webserver.cc:460] Webserver started at http://127.19.139.131:45855/ using document root <none> and password file <none>
I20260928 02:50:51.983174 21481 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:51.983239 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:51.983436 21481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:51.985529 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "b3cda1fca04f48f1b77b24f501d630b1"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "2ba8de81c6d79de9030f0294bacd7429"
server_key_iv: "d3d5744926a542cb786ce5c815c5f60f"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.986084 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "b3cda1fca04f48f1b77b24f501d630b1"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "2ba8de81c6d79de9030f0294bacd7429"
server_key_iv: "d3d5744926a542cb786ce5c815c5f60f"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.990034 21481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:50:51.992517 21496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:51.993906 21481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:50:51.994021 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "b3cda1fca04f48f1b77b24f501d630b1"
format_stamp: "Formatted at 2026-09-28 02:50:51 on dist-test-slave-5vc0"
server_key: "2ba8de81c6d79de9030f0294bacd7429"
server_key_iv: "d3d5744926a542cb786ce5c815c5f60f"
server_key_version: "encryptionkey@0"
I20260928 02:50:51.994128 21481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:52.029505 21481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:52.032956 21481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:52.033146 21481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:52.034622 21481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:52.034677 21481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:52.034729 21481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:52.034755 21481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:52.045185 21481 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:37777
I20260928 02:50:52.045272 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:37777 every 8 connection(s)
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:52.046142 21481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:52.049857 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.035724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:59809 (local address 127.19.139.190:39019)
0928 02:50:52.035856 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.035860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.037039 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 02:50:52.038568 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.038572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.038636 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.038750 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:52.040505 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.041186 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.042914 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.043175 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.043288 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.046210 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:52.046214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:52.046217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:52.046248 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:52.047769 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.048366 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.048369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.048370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.048467 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.048849 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.048851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.048852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.049035 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:52.049108 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.049618 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.049732 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:50:52.050537 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21481
I20260928 02:50:52.050637 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:52.050660 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.036352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.19.139.131:59809)
0928 02:50:52.036873 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.036920 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.038252 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.039018 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.039029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.039666 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:52.040346 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.040362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.041319 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.041323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.042776 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.042788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.043339 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.043344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.043561 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.044274 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:52.044294 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:52.046022 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:52.047918 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:52.047923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:52.047935 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:52.048243 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:52.048579 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:52.048582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:52.048584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:52.048762 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:52.049134 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:52.049140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:52.049380 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.049538 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.049798 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260928 02:50:52.050928 20014 external_mini_cluster.cc:1428] Setting key 0182f4abecfdb7c3292528be90e75e03
I20260928 02:50:52.052053 21612 heartbeater.cc:347] Connected to a master server at 127.19.139.190:39019
I20260928 02:50:52.052227 21612 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:52.052841 21612 heartbeater.cc:511] Master 127.19.139.190:39019 requested a full tablet report, sending...
I20260928 02:50:52.053879 21128 ts_manager.cc:194] Registered new tserver with Master: b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777)
I20260928 02:50:52.054564 21128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:59809
I20260928 02:50:52.061133 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.054246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:38785 (local address 127.19.139.190:39019)
0928 02:50:52.054362 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.054365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.054576 (+ 211us) server_negotiation.cc:407] Connection header received
0928 02:50:52.054684 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.054686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.054729 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.054801 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:52.055276 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.055788 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.056917 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.057211 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.057286 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.058333 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:52.058336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:52.058338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:52.058359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:52.059698 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.060114 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.060116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.060118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.060217 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.060457 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.060460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.060461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.060627 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:52.060694 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.060921 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.061013 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:52.061165 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.054173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.19.139.131:38785)
0928 02:50:52.054494 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.054507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.054599 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.054835 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.054837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.055043 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:52.055167 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.055175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.055914 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.055918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.056804 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.056813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.057326 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.057329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.057400 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.057824 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:52.057834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:52.058199 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:52.059808 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:52.059810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:52.059812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:52.060032 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:52.060289 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:52.060293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:52.060295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:52.060386 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:52.060694 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:52.060698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:52.060789 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.060873 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.061004 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260928 02:50:52.064532 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:52.088624 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.080906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.19.139.190:39019)
0928 02:50:52.081025 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.081028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.081039 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:50:52.081078 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.081081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.081117 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.081183 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:52.081908 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.082415 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.083212 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.083424 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.083509 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.085922 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:52.085925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:52.085927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:52.085953 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:52.087194 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.087581 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.087583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.087584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.087648 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.087835 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.087837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.087838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.088006 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:52.088097 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.088407 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.088502 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:50:52.091504 21128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:52.094587 21128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:52.109378 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.104433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.19.139.130:39273)
0928 02:50:52.104862 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.104872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.105173 (+ 301us) server_negotiation.cc:407] Connection header received
0928 02:50:52.105248 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.105253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.105412 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.105534 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.105729 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.106478 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.107929 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.108676 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.108812 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.108963 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.109117 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.109226 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":83,"threads_started":1}
I20260928 02:50:52.109370 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.103923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39273 (local address 127.0.0.1:44090)
0928 02:50:52.104990 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.105003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.105083 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.105529 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.105532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.105547 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.105629 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.105635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.106601 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.106605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.107806 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.107815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.108801 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.108804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.108911 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.108956 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.109077 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.109210 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":128,"thread_start_us":103,"threads_started":1}
I20260928 02:50:52.109741 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.103192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:44093 (local address 127.0.0.1:45766)
0928 02:50:52.103650 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.103699 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.103828 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.104884 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.104890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.104914 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.105056 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.105064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.106529 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.106533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.108259 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.108270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.109365 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.109368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.109448 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.109475 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.109548 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.109620 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":157,"thread_start_us":79,"threads_started":1}
I20260928 02:50:52.110093 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.103891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.19.139.131:37777)
0928 02:50:52.104259 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.104272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.104545 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:50:52.104650 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.104654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.104797 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.104950 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.105298 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.106405 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.108382 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.109125 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.109255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.109369 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.109853 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.109955 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":126,"threads_started":1}
I20260928 02:50:52.110127 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.103314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.19.139.129:44093)
0928 02:50:52.104391 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.104397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.104420 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:50:52.104494 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.104499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.104725 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.104887 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.105177 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.106400 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.108398 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.109082 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.109182 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.109306 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.109812 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.109939 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":89,"threads_started":1}
I20260928 02:50:52.110527 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.103353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:37777 (local address 127.0.0.1:56490)
0928 02:50:52.104450 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.104461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.104549 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.105070 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.105073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.105086 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.105195 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.105201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.106529 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.106533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.108259 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.108270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.109445 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.109449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.109574 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.109596 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.109659 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.109726 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":222,"thread_start_us":94,"threads_started":1}
I20260928 02:50:52.113173 21266 tablet_service.cc:1467] Processing CreateTablet for tablet 7a22614e89944694a2dc4383374ce65b (DEFAULT_TABLE table=test-table [id=fb8b3c5d4e7b4136ac639a429e5f7ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:52.113174 21390 tablet_service.cc:1467] Processing CreateTablet for tablet 7a22614e89944694a2dc4383374ce65b (DEFAULT_TABLE table=test-table [id=fb8b3c5d4e7b4136ac639a429e5f7ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:52.113333 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 7a22614e89944694a2dc4383374ce65b (DEFAULT_TABLE table=test-table [id=fb8b3c5d4e7b4136ac639a429e5f7ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:52.114291 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a22614e89944694a2dc4383374ce65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.114301 21390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a22614e89944694a2dc4383374ce65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.114301 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a22614e89944694a2dc4383374ce65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.120947 21635 tablet_bootstrap.cc:492] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Bootstrap starting.
I20260928 02:50:52.122983 21636 tablet_bootstrap.cc:492] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Bootstrap starting.
I20260928 02:50:52.123980 21635 tablet_bootstrap.cc:654] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.124454 21637 tablet_bootstrap.cc:492] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Bootstrap starting.
I20260928 02:50:52.124918 21635 log.cc:824] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:52.125240 21636 tablet_bootstrap.cc:654] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.126068 21636 log.cc:824] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:52.126410 21637 tablet_bootstrap.cc:654] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.127170 21635 tablet_bootstrap.cc:492] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: No bootstrap required, opened a new log
I20260928 02:50:52.127206 21637 log.cc:824] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:52.127374 21635 ts_tablet_manager.cc:1397] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:50:52.128139 21636 tablet_bootstrap.cc:492] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: No bootstrap required, opened a new log
I20260928 02:50:52.128437 21636 ts_tablet_manager.cc:1397] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 02:50:52.129227 21637 tablet_bootstrap.cc:492] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: No bootstrap required, opened a new log
I20260928 02:50:52.129400 21637 ts_tablet_manager.cc:1397] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:50:52.131497 21635 raft_consensus.cc:348] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.131908 21635 raft_consensus.cc:374] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.131980 21635 raft_consensus.cc:729] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cda1fca04f48f1b77b24f501d630b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.132397 21636 raft_consensus.cc:348] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.132522 21635 consensus_queue.cc:260] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.132651 21636 raft_consensus.cc:374] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.132726 21636 raft_consensus.cc:729] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b392be2606374e418766369fb68e560f, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.132640 21637 raft_consensus.cc:348] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.132943 21637 raft_consensus.cc:374] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.133037 21637 raft_consensus.cc:729] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fa5e12a298432188fab4d7287c0de5, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.133229 21612 heartbeater.cc:503] Master 127.19.139.190:39019 was elected leader, sending a full tablet report...
I20260928 02:50:52.133185 21636 consensus_queue.cc:260] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.133616 21635 ts_tablet_manager.cc:1428] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:50:52.133562 21637 consensus_queue.cc:260] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.134100 21636 ts_tablet_manager.cc:1428] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 02:50:52.134258 21332 heartbeater.cc:503] Master 127.19.139.190:39019 was elected leader, sending a full tablet report...
I20260928 02:50:52.134409 21637 ts_tablet_manager.cc:1428] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:50:52.134431 21472 heartbeater.cc:503] Master 127.19.139.190:39019 was elected leader, sending a full tablet report...
I20260928 02:50:52.143599 21641 raft_consensus.cc:482] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:52.143805 21641 raft_consensus.cc:504] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.145077 21641 leader_election.cc:288] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093), b392be2606374e418766369fb68e560f (127.19.139.130:39273)
I20260928 02:50:52.148489 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.145421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:33791 (local address 127.19.139.129:44093)
0928 02:50:52.145543 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.145547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.145562 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:50:52.145616 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.145620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.145664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.145741 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.146019 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.146519 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.147521 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.147940 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.148046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.148102 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.148286 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.148358 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:52.148489 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.145296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:44093 (local address 127.19.139.131:33791)
0928 02:50:52.145414 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.145429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.145515 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.145770 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.145773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.145792 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.145911 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.145917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.146620 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.146623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.147398 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.147406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.148046 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.148050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.148144 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.148196 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.148271 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.148347 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:50:52.149173 21286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a22614e89944694a2dc4383374ce65b" candidate_uuid: "b3cda1fca04f48f1b77b24f501d630b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fa5e12a298432188fab4d7287c0de5" is_pre_election: true
I20260928 02:50:52.149290 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.145556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39273 (local address 127.19.139.131:56593)
0928 02:50:52.145964 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.145974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.146058 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.146251 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.146254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.146268 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.146387 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.146392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.147099 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.147103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.148263 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.148273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.148983 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.148986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.149067 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.149083 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.149138 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.149186 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":140,"threads_started":1}
I20260928 02:50:52.149346 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.145706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:56593 (local address 127.19.139.130:39273)
0928 02:50:52.145832 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.145836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.146045 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:50:52.146135 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.146138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.146179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.146236 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.146488 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.147004 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.148379 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.148862 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.148989 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.149034 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.149165 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.149234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:50:52.149529 21286 raft_consensus.cc:2454] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cda1fca04f48f1b77b24f501d630b1 in term 0.
I20260928 02:50:52.149865 21426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a22614e89944694a2dc4383374ce65b" candidate_uuid: "b3cda1fca04f48f1b77b24f501d630b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b392be2606374e418766369fb68e560f" is_pre_election: true
I20260928 02:50:52.150003 21498 leader_election.cc:302] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3cda1fca04f48f1b77b24f501d630b1, b6fa5e12a298432188fab4d7287c0de5; no voters:
I20260928 02:50:52.150156 21426 raft_consensus.cc:2454] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cda1fca04f48f1b77b24f501d630b1 in term 0.
I20260928 02:50:52.150300 21641 raft_consensus.cc:2781] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:52.150420 21641 raft_consensus.cc:482] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:52.150499 21641 raft_consensus.cc:3037] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.151571 21641 raft_consensus.cc:504] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.151902 21641 leader_election.cc:288] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [CANDIDATE]: Term 1 election: Requested vote from peers b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093), b392be2606374e418766369fb68e560f (127.19.139.130:39273)
I20260928 02:50:52.152267 21286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a22614e89944694a2dc4383374ce65b" candidate_uuid: "b3cda1fca04f48f1b77b24f501d630b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fa5e12a298432188fab4d7287c0de5"
I20260928 02:50:52.152325 21426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a22614e89944694a2dc4383374ce65b" candidate_uuid: "b3cda1fca04f48f1b77b24f501d630b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b392be2606374e418766369fb68e560f"
I20260928 02:50:52.152390 21286 raft_consensus.cc:3037] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.152431 21426 raft_consensus.cc:3037] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.153627 21286 raft_consensus.cc:2454] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cda1fca04f48f1b77b24f501d630b1 in term 1.
I20260928 02:50:52.153875 21426 raft_consensus.cc:2454] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cda1fca04f48f1b77b24f501d630b1 in term 1.
I20260928 02:50:52.153924 21498 leader_election.cc:302] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3cda1fca04f48f1b77b24f501d630b1, b6fa5e12a298432188fab4d7287c0de5; no voters:
I20260928 02:50:52.154068 21641 raft_consensus.cc:2781] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:52.154331 21641 raft_consensus.cc:686] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 1 LEADER]: Becoming Leader. State: Replica: b3cda1fca04f48f1b77b24f501d630b1, State: Running, Role: LEADER
I20260928 02:50:52.154853 21641 consensus_queue.cc:237] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.157990 21128 catalog_manager.cc:5495] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131). New cstate: current_term: 1 leader_uuid: "b3cda1fca04f48f1b77b24f501d630b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } health_report { overall_health: HEALTHY } } }
I20260928 02:50:52.182214 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.178034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.19.139.131:37777)
0928 02:50:52.178167 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.178172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.178186 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:52.178368 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.178370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.178425 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.178507 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:52.178833 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.179405 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.180474 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.180782 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.180903 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.181282 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:52.181775 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:52.181892 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.182000 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.182071 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:52.188272 21286 raft_consensus.cc:1261] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Refusing update from remote peer b3cda1fca04f48f1b77b24f501d630b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:52.188272 21426 raft_consensus.cc:1261] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Refusing update from remote peer b3cda1fca04f48f1b77b24f501d630b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:52.189023 21645 consensus_queue.cc:1035] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:52.189219 21641 consensus_queue.cc:1035] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:52.210386 21127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:52.210609 21127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:52.213202 21127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: Sending DeleteTablet for 3 replicas of tablet 7a22614e89944694a2dc4383374ce65b
I20260928 02:50:52.214092 21528 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a22614e89944694a2dc4383374ce65b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:52 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:56490
I20260928 02:50:52.214263 21266 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a22614e89944694a2dc4383374ce65b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:52 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:45766
I20260928 02:50:52.214267 21390 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a22614e89944694a2dc4383374ce65b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:52 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:44090
I20260928 02:50:52.214702 21659 tablet_replica.cc:330] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: stopping tablet replica
I20260928 02:50:52.214987 21659 raft_consensus.cc:2229] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:52.215312 21659 raft_consensus.cc:2258] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:52.215314 21660 tablet_replica.cc:330] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: stopping tablet replica
I20260928 02:50:52.215566 21127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:50:52.215756 21658 tablet_replica.cc:330] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: stopping tablet replica
I20260928 02:50:52.216008 21658 raft_consensus.cc:2229] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 1 LEADER]: Raft consensus shutting down.
W20260928 02:50:52.216030 21127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:52.216360 21658 raft_consensus.cc:2258] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:52.217185 21660 raft_consensus.cc:2229] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:52.217720 21659 ts_tablet_manager.cc:1905] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:52.218428 21660 raft_consensus.cc:2258] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:52.218977 21658 ts_tablet_manager.cc:1905] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:52.220549 21659 ts_tablet_manager.cc:1918] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:52.220685 21659 log.cc:1196] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/7a22614e89944694a2dc4383374ce65b
I20260928 02:50:52.221011 21659 ts_tablet_manager.cc:1939] T 7a22614e89944694a2dc4383374ce65b P b6fa5e12a298432188fab4d7287c0de5: Deleting consensus metadata
I20260928 02:50:52.221676 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4581eabb784fa6af95c7f0e8c54cab (DEFAULT_TABLE table=test-table [id=2f2e79270a74408494c3aac54e6ef00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:52.221697 21266 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4581eabb784fa6af95c7f0e8c54cab (DEFAULT_TABLE table=test-table [id=2f2e79270a74408494c3aac54e6ef00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:52.221921 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4581eabb784fa6af95c7f0e8c54cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.221956 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4581eabb784fa6af95c7f0e8c54cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.223408 21660 ts_tablet_manager.cc:1905] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:52.223999 21637 tablet_bootstrap.cc:492] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Bootstrap starting.
I20260928 02:50:52.224216 21390 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4581eabb784fa6af95c7f0e8c54cab (DEFAULT_TABLE table=test-table [id=2f2e79270a74408494c3aac54e6ef00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:52.224464 21390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4581eabb784fa6af95c7f0e8c54cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.224974 21637 tablet_bootstrap.cc:654] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.225797 21660 ts_tablet_manager.cc:1918] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:52.225872 21660 log.cc:1196] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/7a22614e89944694a2dc4383374ce65b
I20260928 02:50:52.226157 21660 ts_tablet_manager.cc:1939] T 7a22614e89944694a2dc4383374ce65b P b392be2606374e418766369fb68e560f: Deleting consensus metadata
I20260928 02:50:52.226280 21637 tablet_bootstrap.cc:492] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: No bootstrap required, opened a new log
I20260928 02:50:52.226382 21637 ts_tablet_manager.cc:1397] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:52.226576 21635 tablet_bootstrap.cc:492] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Bootstrap starting.
I20260928 02:50:52.226971 21637 raft_consensus.cc:348] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.227077 21637 raft_consensus.cc:374] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.227104 21637 raft_consensus.cc:729] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fa5e12a298432188fab4d7287c0de5, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.227209 21637 consensus_queue.cc:260] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.227454 21637 ts_tablet_manager.cc:1428] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:52.227505 21635 tablet_bootstrap.cc:654] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.227991 21636 tablet_bootstrap.cc:492] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Bootstrap starting.
I20260928 02:50:52.228456 21114 catalog_manager.cc:4841] TS b392be2606374e418766369fb68e560f (127.19.139.130:39273): tablet 7a22614e89944694a2dc4383374ce65b (table test-table [id=fb8b3c5d4e7b4136ac639a429e5f7ae6]) successfully deleted
I20260928 02:50:52.228606 21635 tablet_bootstrap.cc:492] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: No bootstrap required, opened a new log
I20260928 02:50:52.228673 21635 ts_tablet_manager.cc:1397] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:52.228904 21636 tablet_bootstrap.cc:654] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.229095 21635 raft_consensus.cc:348] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.229195 21635 raft_consensus.cc:374] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.229223 21635 raft_consensus.cc:729] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cda1fca04f48f1b77b24f501d630b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.229251 21114 catalog_manager.cc:4841] TS b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093): tablet 7a22614e89944694a2dc4383374ce65b (table test-table [id=fb8b3c5d4e7b4136ac639a429e5f7ae6]) successfully deleted
I20260928 02:50:52.229353 21635 consensus_queue.cc:260] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.229597 21635 ts_tablet_manager.cc:1428] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:52.230475 21658 ts_tablet_manager.cc:1918] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:52.230618 21658 log.cc:1196] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/7a22614e89944694a2dc4383374ce65b
I20260928 02:50:52.230976 21658 ts_tablet_manager.cc:1939] T 7a22614e89944694a2dc4383374ce65b P b3cda1fca04f48f1b77b24f501d630b1: Deleting consensus metadata
I20260928 02:50:52.232213 21636 tablet_bootstrap.cc:492] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: No bootstrap required, opened a new log
I20260928 02:50:52.232244 21115 catalog_manager.cc:4841] TS b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777): tablet 7a22614e89944694a2dc4383374ce65b (table test-table [id=fb8b3c5d4e7b4136ac639a429e5f7ae6]) successfully deleted
I20260928 02:50:52.232299 21636 ts_tablet_manager.cc:1397] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:50:52.232728 21636 raft_consensus.cc:348] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.232843 21636 raft_consensus.cc:374] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.232880 21636 raft_consensus.cc:729] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b392be2606374e418766369fb68e560f, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.233009 21636 consensus_queue.cc:260] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.233249 21636 ts_tablet_manager.cc:1428] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:52.254367 21642 raft_consensus.cc:482] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:52.254518 21642 raft_consensus.cc:504] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.255592 21642 leader_election.cc:288] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093), b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777)
I20260928 02:50:52.260197 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.256285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:48755 (local address 127.19.139.131:37777)
0928 02:50:52.256401 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.256405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.256421 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:50:52.256477 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.256480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.256525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.256604 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.256840 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.257592 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.258903 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.259410 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.259514 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.259568 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.259999 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.260070 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:50:52.260200 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.255794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:44093 (local address 127.19.139.130:54791)
0928 02:50:52.256014 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.256030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.256123 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.256541 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.256543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.256557 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.256671 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.256678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.257444 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.257448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.258662 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.258673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.259661 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.259665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.259799 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.259864 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.259939 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.260017 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":27}
I20260928 02:50:52.260226 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.255835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:37777 (local address 127.19.139.130:48755)
0928 02:50:52.256301 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.256313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.256405 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.256615 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.256618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.256631 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:52.256724 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.256730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.257699 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.257703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.258791 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.258799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.259644 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.259660 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.259795 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.259852 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.259962 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.260037 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":40,"thread_start_us":120,"threads_started":1}
I20260928 02:50:52.260720 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.256004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:54791 (local address 127.19.139.129:44093)
0928 02:50:52.256112 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.256116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.256130 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:52.256299 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.256303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.256359 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.256433 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:52.256814 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.257344 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.258791 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.259434 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.259551 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.259600 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.260549 (+ 949us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.260629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:52.260973 21566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4581eabb784fa6af95c7f0e8c54cab" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cda1fca04f48f1b77b24f501d630b1" is_pre_election: true
I20260928 02:50:52.261132 21286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4581eabb784fa6af95c7f0e8c54cab" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fa5e12a298432188fab4d7287c0de5" is_pre_election: true
I20260928 02:50:52.261235 21566 raft_consensus.cc:2454] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 0.
I20260928 02:50:52.261279 21286 raft_consensus.cc:2454] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 0.
I20260928 02:50:52.261693 21359 leader_election.cc:302] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b392be2606374e418766369fb68e560f, b3cda1fca04f48f1b77b24f501d630b1; no voters:
I20260928 02:50:52.262004 21642 raft_consensus.cc:2781] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:52.262071 21642 raft_consensus.cc:482] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:52.262116 21642 raft_consensus.cc:3037] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.263366 21642 raft_consensus.cc:504] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.263741 21642 leader_election.cc:288] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 election: Requested vote from peers b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093), b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777)
I20260928 02:50:52.264066 21286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4581eabb784fa6af95c7f0e8c54cab" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fa5e12a298432188fab4d7287c0de5"
I20260928 02:50:52.264262 21286 raft_consensus.cc:3037] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.264299 21566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4581eabb784fa6af95c7f0e8c54cab" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cda1fca04f48f1b77b24f501d630b1"
I20260928 02:50:52.264464 21566 raft_consensus.cc:3037] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.265373 21566 raft_consensus.cc:2454] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 1.
I20260928 02:50:52.265384 21286 raft_consensus.cc:2454] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 1.
I20260928 02:50:52.265645 21359 leader_election.cc:302] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b392be2606374e418766369fb68e560f, b3cda1fca04f48f1b77b24f501d630b1; no voters:
I20260928 02:50:52.265794 21642 raft_consensus.cc:2781] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:52.266028 21642 raft_consensus.cc:686] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 1 LEADER]: Becoming Leader. State: Replica: b392be2606374e418766369fb68e560f, State: Running, Role: LEADER
I20260928 02:50:52.266314 21642 consensus_queue.cc:237] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.268962 21126 catalog_manager.cc:5495] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f reported cstate change: term changed from 0 to 1, leader changed from <none> to b392be2606374e418766369fb68e560f (127.19.139.130). New cstate: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } health_report { overall_health: UNKNOWN } } }
W20260928 02:50:52.297320 21613 tablet.cc:2369] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:52.314914 21473 tablet.cc:2369] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:52.325374 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.310157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.19.139.190:39019)
0928 02:50:52.310293 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.310298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.310423 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:50:52.310528 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.310532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.310582 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.310673 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:52.311401 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.312139 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.313388 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.313734 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.313839 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.321421 (+ 7582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:52.321425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:52.321428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:52.321460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:52.323643 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.324182 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.324185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.324187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.324275 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.324505 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.324508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.324509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.324704 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:52.324762 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.325084 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.325201 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:50:52.326488 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.310053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:39019 (local address 127.0.0.1:41890)
0928 02:50:52.310326 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.310342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.310446 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.310725 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.310727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.311109 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:52.311277 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.311287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.312316 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.312320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.313232 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.313243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.313856 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.313860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.313952 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.316941 (+ 2989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:52.316958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:52.321247 (+ 4289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:52.323773 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:52.323783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:52.323790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:52.324067 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:52.324361 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:52.324364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:52.324366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:52.324433 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:52.324821 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:52.324828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:52.324937 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.326149 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.326329 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260928 02:50:52.331184 21126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:41890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:50:52.332074 21126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:52.337584 21266 tablet_service.cc:1467] Processing CreateTablet for tablet 924e6e5f75844d3fb44aeb07e796a7e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=493129bdf5a44f088044c3d41b408b52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:50:52.338033 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924e6e5f75844d3fb44aeb07e796a7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.340580 21390 tablet_service.cc:1467] Processing CreateTablet for tablet 924e6e5f75844d3fb44aeb07e796a7e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=493129bdf5a44f088044c3d41b408b52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:50:52.340984 21390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924e6e5f75844d3fb44aeb07e796a7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:52.343961 21636 tablet_bootstrap.cc:492] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: Bootstrap starting.
I20260928 02:50:52.345038 21636 tablet_bootstrap.cc:654] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.346015 21528 tablet_service.cc:1467] Processing CreateTablet for tablet 924e6e5f75844d3fb44aeb07e796a7e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=493129bdf5a44f088044c3d41b408b52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:50:52.346396 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924e6e5f75844d3fb44aeb07e796a7e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260928 02:50:52.348997 21333 tablet.cc:2369] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:52.349258 21635 tablet_bootstrap.cc:492] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: Bootstrap starting.
I20260928 02:50:52.350607 21635 tablet_bootstrap.cc:654] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.352056 21635 tablet_bootstrap.cc:492] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: No bootstrap required, opened a new log
I20260928 02:50:52.352136 21635 ts_tablet_manager.cc:1397] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:52.352675 21635 raft_consensus.cc:348] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.352789 21635 raft_consensus.cc:374] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.352823 21635 raft_consensus.cc:729] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cda1fca04f48f1b77b24f501d630b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.352963 21635 consensus_queue.cc:260] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.353304 21635 ts_tablet_manager.cc:1428] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:52.353796 21653 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.353950 21653 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.356328 21637 tablet_bootstrap.cc:492] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: Bootstrap starting.
I20260928 02:50:52.357450 21637 tablet_bootstrap.cc:654] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:52.358760 21636 tablet_bootstrap.cc:492] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: No bootstrap required, opened a new log
I20260928 02:50:52.358769 21637 tablet_bootstrap.cc:492] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: No bootstrap required, opened a new log
I20260928 02:50:52.358857 21637 ts_tablet_manager.cc:1397] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:52.358856 21636 ts_tablet_manager.cc:1397] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260928 02:50:52.359315 21636 raft_consensus.cc:348] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.359321 21637 raft_consensus.cc:348] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.359418 21636 raft_consensus.cc:374] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.359431 21637 raft_consensus.cc:374] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:52.359447 21636 raft_consensus.cc:729] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b392be2606374e418766369fb68e560f, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.359467 21637 raft_consensus.cc:729] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fa5e12a298432188fab4d7287c0de5, State: Initialized, Role: FOLLOWER
I20260928 02:50:52.359633 21636 consensus_queue.cc:260] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.359622 21637 consensus_queue.cc:260] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.359900 21637 ts_tablet_manager.cc:1428] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:52.359891 21643 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.360069 21643 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.360364 21642 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.360499 21642 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.360792 21636 ts_tablet_manager.cc:1428] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:52.477156 21642 raft_consensus.cc:482] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:52.477304 21642 raft_consensus.cc:504] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.477682 21642 leader_election.cc:288] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093), b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777)
I20260928 02:50:52.478194 21286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924e6e5f75844d3fb44aeb07e796a7e6" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fa5e12a298432188fab4d7287c0de5" is_pre_election: true
I20260928 02:50:52.478196 21566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924e6e5f75844d3fb44aeb07e796a7e6" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cda1fca04f48f1b77b24f501d630b1" is_pre_election: true
I20260928 02:50:52.478354 21286 raft_consensus.cc:2454] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 0.
I20260928 02:50:52.478385 21566 raft_consensus.cc:2454] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 0.
I20260928 02:50:52.478678 21358 leader_election.cc:302] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b392be2606374e418766369fb68e560f, b6fa5e12a298432188fab4d7287c0de5; no voters:
I20260928 02:50:52.478875 21642 raft_consensus.cc:2781] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:52.478951 21642 raft_consensus.cc:482] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:52.478998 21642 raft_consensus.cc:3037] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.480091 21642 raft_consensus.cc:504] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.480454 21642 leader_election.cc:288] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 election: Requested vote from peers b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093), b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777)
I20260928 02:50:52.480808 21566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924e6e5f75844d3fb44aeb07e796a7e6" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cda1fca04f48f1b77b24f501d630b1"
I20260928 02:50:52.480836 21286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924e6e5f75844d3fb44aeb07e796a7e6" candidate_uuid: "b392be2606374e418766369fb68e560f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6fa5e12a298432188fab4d7287c0de5"
I20260928 02:50:52.480930 21566 raft_consensus.cc:3037] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.480963 21286 raft_consensus.cc:3037] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:52.481880 21566 raft_consensus.cc:2454] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 1.
I20260928 02:50:52.481932 21286 raft_consensus.cc:2454] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b392be2606374e418766369fb68e560f in term 1.
I20260928 02:50:52.482162 21359 leader_election.cc:302] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b392be2606374e418766369fb68e560f, b3cda1fca04f48f1b77b24f501d630b1; no voters:
I20260928 02:50:52.482313 21642 raft_consensus.cc:2781] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:52.482409 21642 raft_consensus.cc:686] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [term 1 LEADER]: Becoming Leader. State: Replica: b392be2606374e418766369fb68e560f, State: Running, Role: LEADER
I20260928 02:50:52.482542 21642 consensus_queue.cc:237] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } }
I20260928 02:50:52.483198 21665 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: TxnStatusTablet state changed. Reason: New leader b392be2606374e418766369fb68e560f. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.483301 21665 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:50:52.483520 21677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:50:52.483687 21677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:50:52.484334 21126 catalog_manager.cc:5495] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f reported cstate change: term changed from 0 to 1, leader changed from <none> to b392be2606374e418766369fb68e560f (127.19.139.130). New cstate: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:52.499261 21672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.495010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39273 (local address 127.0.0.1:44096)
0928 02:50:52.495163 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.495178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.495274 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.495582 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.495584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.495590 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:50:52.495680 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.495683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.496716 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.496718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.497356 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.497366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.498009 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.498013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.498083 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.498133 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:50:52.498999 (+ 866us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:50:52.499015 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.499067 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.499140 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:50:52.499297 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.495115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.19.139.130:39273)
0928 02:50:52.495241 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.495245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.495262 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:50:52.495403 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.495406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.495473 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.495576 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:52.495768 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.496609 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.497486 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.497866 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.497973 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.498333 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:52.498889 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:52.499005 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.499080 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.499139 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:50:52.502300 21286 raft_consensus.cc:1261] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Refusing update from remote peer b392be2606374e418766369fb68e560f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:52.502401 21566 raft_consensus.cc:1261] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Refusing update from remote peer b392be2606374e418766369fb68e560f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:52.502971 21665 consensus_queue.cc:1035] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:52.503340 21642 consensus_queue.cc:1035] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:52.505403 21653 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: TxnStatusTablet state changed. Reason: New leader b392be2606374e418766369fb68e560f. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.505507 21653 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.505553 21643 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: TxnStatusTablet state changed. Reason: New leader b392be2606374e418766369fb68e560f. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.505676 21643 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.507454 21665 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.507567 21665 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:50:52.507916 21649 mvcc.cc:205] Tried to move back new op lower bound from 7334149539844272128 to 7334149539769929728. Current Snapshot: MvccSnapshot[applied={T|T < 7334149539844272128}]
I20260928 02:50:52.508083 21643 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.508198 21643 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b6fa5e12a298432188fab4d7287c0de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.511020 21665 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.511135 21665 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b392be2606374e418766369fb68e560f: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:50:52.511747 21648 mvcc.cc:205] Tried to move back new op lower bound from 7334149539844272128 to 7334149539769929728. Current Snapshot: MvccSnapshot[applied={T|T < 7334149539844272128}]
I20260928 02:50:52.511936 21653 tablet_replica.cc:439] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b392be2606374e418766369fb68e560f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 } } peers { permanent_uuid: "b392be2606374e418766369fb68e560f" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39273 } } peers { permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 } } }
I20260928 02:50:52.512032 21653 tablet_replica.cc:442] T 924e6e5f75844d3fb44aeb07e796a7e6 P b3cda1fca04f48f1b77b24f501d630b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:52.515322 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.512377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.19.139.130:39273)
0928 02:50:52.512507 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.512510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.512523 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:52.512638 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.512641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.512688 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.512748 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:52.513031 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.513524 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.514420 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.514629 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.514708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.514942 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:52.515006 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:52.515061 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.515152 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.515201 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:50:52.525835 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.522295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:50291 (local address 127.19.139.130:39273)
0928 02:50:52.522407 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.522411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.522579 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:50:52.522696 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.522699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.522736 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.522791 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:52.523054 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.523538 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.524823 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.525045 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.525131 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.525428 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:52.525497 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:52.525554 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.525652 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.525717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":39}
I20260928 02:50:52.525970 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.522235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39273 (local address 127.19.139.130:50291)
0928 02:50:52.522503 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:52.522517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:52.522626 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:52.522809 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:52.522812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:52.522822 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:50:52.522976 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.522982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.523620 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.523623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.524660 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:52.524670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.525145 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.525149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:52.525239 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.525289 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:50:52.525574 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:50:52.525580 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:50:52.525646 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:50:52.525755 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":37,"thread_start_us":101,"threads_started":1}
I20260928 02:50:52.532773 21566 raft_consensus.cc:1261] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Refusing update from remote peer b392be2606374e418766369fb68e560f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:52.532881 21286 raft_consensus.cc:1261] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Refusing update from remote peer b392be2606374e418766369fb68e560f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:52.533119 21665 consensus_queue.cc:1035] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cda1fca04f48f1b77b24f501d630b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 37777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:52.533262 21682 consensus_queue.cc:1035] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fa5e12a298432188fab4d7287c0de5" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:52.577466 21126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:52.577620 21126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:52.579702 21126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ae2f68ac6834e3790ec44811074cdbf: Sending DeleteTablet for 3 replicas of tablet 2a4581eabb784fa6af95c7f0e8c54cab
I20260928 02:50:52.580268 21405 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4581eabb784fa6af95c7f0e8c54cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:52 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:44090
I20260928 02:50:52.580348 21266 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4581eabb784fa6af95c7f0e8c54cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:52 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:45766
I20260928 02:50:52.580458 21660 tablet_replica.cc:330] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: stopping tablet replica
I20260928 02:50:52.580529 21659 tablet_replica.cc:330] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: stopping tablet replica
I20260928 02:50:52.580561 21660 raft_consensus.cc:2229] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:52.580626 21659 raft_consensus.cc:2229] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:52.580735 21659 raft_consensus.cc:2258] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:52.580802 21660 raft_consensus.cc:2258] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:52.580919 21528 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4581eabb784fa6af95c7f0e8c54cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:52 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:56490
I20260928 02:50:52.581090 21658 tablet_replica.cc:330] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: stopping tablet replica
I20260928 02:50:52.581184 21658 raft_consensus.cc:2229] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:52.581290 21658 raft_consensus.cc:2258] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:52.581621 21660 ts_tablet_manager.cc:1905] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:52.582115 21658 ts_tablet_manager.cc:1905] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:52.583938 21660 ts_tablet_manager.cc:1918] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:52.583997 21660 log.cc:1196] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/2a4581eabb784fa6af95c7f0e8c54cab
I20260928 02:50:52.584221 21660 ts_tablet_manager.cc:1939] T 2a4581eabb784fa6af95c7f0e8c54cab P b392be2606374e418766369fb68e560f: Deleting consensus metadata
I20260928 02:50:52.585003 21658 ts_tablet_manager.cc:1918] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:52.585230 21114 catalog_manager.cc:4841] TS b392be2606374e418766369fb68e560f (127.19.139.130:39273): tablet 2a4581eabb784fa6af95c7f0e8c54cab (table test-table [id=2f2e79270a74408494c3aac54e6ef00d]) successfully deleted
I20260928 02:50:52.586259 21658 log.cc:1196] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/2a4581eabb784fa6af95c7f0e8c54cab
I20260928 02:50:52.586532 21658 ts_tablet_manager.cc:1939] T 2a4581eabb784fa6af95c7f0e8c54cab P b3cda1fca04f48f1b77b24f501d630b1: Deleting consensus metadata
I20260928 02:50:52.587185 21659 ts_tablet_manager.cc:1905] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:52.587558 21115 catalog_manager.cc:4841] TS b3cda1fca04f48f1b77b24f501d630b1 (127.19.139.131:37777): tablet 2a4581eabb784fa6af95c7f0e8c54cab (table test-table [id=2f2e79270a74408494c3aac54e6ef00d]) successfully deleted
I20260928 02:50:52.589975 21659 ts_tablet_manager.cc:1918] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:52.590056 21659 log.cc:1196] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/2a4581eabb784fa6af95c7f0e8c54cab
I20260928 02:50:52.590250 21659 ts_tablet_manager.cc:1939] T 2a4581eabb784fa6af95c7f0e8c54cab P b6fa5e12a298432188fab4d7287c0de5: Deleting consensus metadata
I20260928 02:50:52.591048 21114 catalog_manager.cc:4841] TS b6fa5e12a298432188fab4d7287c0de5 (127.19.139.129:44093): tablet 2a4581eabb784fa6af95c7f0e8c54cab (table test-table [id=2f2e79270a74408494c3aac54e6ef00d]) successfully deleted
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:50:52.604243 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.586501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.19.139.129:44093)
0928 02:50:52.586627 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.586632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.588499 (+ 1867us) server_negotiation.cc:407] Connection header received
0928 02:50:52.588657 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.588660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.588730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.588808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:52.589652 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.590369 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.591456 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.591760 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.591865 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.599778 (+ 7913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:52.599791 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:52.599806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:52.599846 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:52.601958 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.602551 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.602559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.602561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.602671 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.603039 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.603043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.603045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.603385 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:52.603596 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.603841 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.604026 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 02:50:52.604707 21326 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778
I20260928 02:50:52.615695 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:52.606510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.19.139.190:39019)
0928 02:50:52.606632 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:52.606636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:52.607079 (+ 443us) server_negotiation.cc:407] Connection header received
0928 02:50:52.607184 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:52.607188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:52.607240 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:52.607324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:52.608120 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.609006 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.610109 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:52.610494 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:52.610591 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:52.612319 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:52.612323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:52.612325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:52.612358 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:52.613959 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.614470 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.614473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.614475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.614556 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:52.614834 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:52.614836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:52.614838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:52.615045 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:52.615162 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:52.615425 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:50:52.615548 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:50:52.616483 21126 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41896
I20260928 02:50:52.617954 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21201
I20260928 02:50:52.623380 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21341
I20260928 02:50:52.630815 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21481
I20260928 02:50:52.636207 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21097
2026-09-28T02:50:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2577 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21696](info): set up 2 sockets
Sep 28 02:50:52 dist-test-slave-5vc0 krb5kdc[21696](info): commencing operation
krb5kdc: starting...
W20260928 02:50:53.587770 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:50:53 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:50:53.710683 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35031
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:53.821753 21709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:53.822046 21709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:53.822088 21709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:53.826171 21709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:53.826242 21709 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:50:53.826259 21709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:53.826275 21709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:53.826289 21709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:53.826303 21709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:53.831718 21709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:35031
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:53.832883 21709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:53.833956 21709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:53.842847 21717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:53.842854 21714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:53.842941 21709 server_base.cc:1034] running on GCE node
W20260928 02:50:53.842852 21715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:53.843719 21709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:53.844971 21709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:53.846141 21709 hybrid_clock.cc:648] HybridClock initialized: now 1790563853846127 us; error 36 us; skew 500 ppm
Sep 28 02:50:53 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563853, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:53.849016 21709 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:50:53.850000 21709 webserver.cc:460] Webserver started at http://127.19.139.190:35005/ using document root <none> and password file <none>
I20260928 02:50:53.850558 21709 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:53.850625 21709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:53.850819 21709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:53.853025 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "a81b9f3350774968bf8148e9f3f3f528"
format_stamp: "Formatted at 2026-09-28 02:50:53 on dist-test-slave-5vc0"
server_key: "4ff779d7215c9d91b24d6fa26062f38e"
server_key_iv: "0adc59c7d57902b1773e8145b7b4533d"
server_key_version: "encryptionkey@0"
I20260928 02:50:53.853567 21709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "a81b9f3350774968bf8148e9f3f3f528"
format_stamp: "Formatted at 2026-09-28 02:50:53 on dist-test-slave-5vc0"
server_key: "4ff779d7215c9d91b24d6fa26062f38e"
server_key_iv: "0adc59c7d57902b1773e8145b7b4533d"
server_key_version: "encryptionkey@0"
I20260928 02:50:53.857363 21709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:50:53.859778 21724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:53.861066 21709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:53.861192 21709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "a81b9f3350774968bf8148e9f3f3f528"
format_stamp: "Formatted at 2026-09-28 02:50:53 on dist-test-slave-5vc0"
server_key: "4ff779d7215c9d91b24d6fa26062f38e"
server_key_iv: "0adc59c7d57902b1773e8145b7b4533d"
server_key_version: "encryptionkey@0"
I20260928 02:50:53.861296 21709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:53.884588 21709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:53.887784 21709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:53.887969 21709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:53.895718 21709 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:35031
I20260928 02:50:53.895736 21786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:35031 every 8 connection(s)
I20260928 02:50:53.896747 21709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:53.897018 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21709
I20260928 02:50:53.897118 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:50:53.897403 20014 external_mini_cluster.cc:1428] Setting key 65dd53fd0b76b7bb986745884a48d9a4
I20260928 02:50:53.899922 21787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:53.905539 21787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Bootstrap starting.
Sep 28 02:50:53 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:53.907805 21787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:53.908880 21787 log.cc:824] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:53.910789 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:53.898830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.19.139.190:35031)
0928 02:50:53.899249 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:53.899264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:53.899299 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:53.899954 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:53.899974 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:53.900376 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:53.900691 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:53.901456 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:53.902437 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:53.903269 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:53.903745 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:53.903886 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:53.906423 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:53.906427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:53.906441 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:53.906468 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:53.908029 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:53.908464 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:53.908469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:53.908474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:53.908583 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:53.908831 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:53.908835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:53.908837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:53.909276 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:53.909486 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:53.909770 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:50:53.910068 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260928 02:50:53.910939 21787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: No bootstrap required, opened a new log
I20260928 02:50:53.914233 21787 raft_consensus.cc:348] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81b9f3350774968bf8148e9f3f3f528" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35031 } }
I20260928 02:50:53.914469 21787 raft_consensus.cc:374] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:53.914531 21787 raft_consensus.cc:729] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81b9f3350774968bf8148e9f3f3f528, State: Initialized, Role: FOLLOWER
I20260928 02:50:53.914964 21787 consensus_queue.cc:260] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81b9f3350774968bf8148e9f3f3f528" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35031 } }
I20260928 02:50:53.915092 21787 raft_consensus.cc:388] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:53.915150 21787 raft_consensus.cc:482] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:53.915244 21787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:53.916306 21787 raft_consensus.cc:504] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81b9f3350774968bf8148e9f3f3f528" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35031 } }
I20260928 02:50:53.916635 21787 leader_election.cc:302] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a81b9f3350774968bf8148e9f3f3f528; no voters:
I20260928 02:50:53.916935 21787 leader_election.cc:288] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:53.917053 21792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:53.917253 21792 raft_consensus.cc:686] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [term 1 LEADER]: Becoming Leader. State: Replica: a81b9f3350774968bf8148e9f3f3f528, State: Running, Role: LEADER
I20260928 02:50:53.917589 21792 consensus_queue.cc:237] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81b9f3350774968bf8148e9f3f3f528" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35031 } }
I20260928 02:50:53.918109 21787 sys_catalog.cc:564] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:53.919090 21793 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a81b9f3350774968bf8148e9f3f3f528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81b9f3350774968bf8148e9f3f3f528" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35031 } } }
I20260928 02:50:53.919217 21793 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:53.919601 21801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:53.919569 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a81b9f3350774968bf8148e9f3f3f528. Latest consensus state: current_term: 1 leader_uuid: "a81b9f3350774968bf8148e9f3f3f528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a81b9f3350774968bf8148e9f3f3f528" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35031 } } }
I20260928 02:50:53.919819 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:53.922662 21801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:53.927717 21801 catalog_manager.cc:1347] Generated new cluster ID: 545671256bc14102a5ee4836bb3d5520
I20260928 02:50:53.927790 21801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:53.942828 21801 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:53.943822 21801 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:53.952312 21801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Generated new TSK 0
I20260928 02:50:53.953174 21801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:50:53.990870 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35031
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:54.103650 21813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:54.103956 21813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:54.104005 21813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:54.108326 21813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:54.108404 21813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:54.108448 21813 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:50:54.108461 21813 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:50:54.108500 21813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:50:54.114056 21813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:54.115321 21813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:54.116526 21813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:54.126636 21819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:54.126638 21818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:54.126636 21821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:54.126700 21813 server_base.cc:1034] running on GCE node
I20260928 02:50:54.127262 21813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:54.127928 21813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:54.129132 21813 hybrid_clock.cc:648] HybridClock initialized: now 1790563854129103 us; error 39 us; skew 500 ppm
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:54.131995 21813 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:50:54.132948 21813 webserver.cc:460] Webserver started at http://127.19.139.129:37001/ using document root <none> and password file <none>
I20260928 02:50:54.133496 21813 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:54.133566 21813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:54.133765 21813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:54.135797 21813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "a4bc8283026d4b30bf0c473c6dbb59fa"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "0aa0544f2719bdfb66b6d2cfdbef2c9b"
server_key_iv: "07e3333f020d3cc496bbdac355e81952"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.136389 21813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "a4bc8283026d4b30bf0c473c6dbb59fa"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "0aa0544f2719bdfb66b6d2cfdbef2c9b"
server_key_iv: "07e3333f020d3cc496bbdac355e81952"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.140260 21813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:50:54.142547 21828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.143793 21813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:54.143896 21813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "a4bc8283026d4b30bf0c473c6dbb59fa"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "0aa0544f2719bdfb66b6d2cfdbef2c9b"
server_key_iv: "07e3333f020d3cc496bbdac355e81952"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.144009 21813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:54.168622 21813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:54.171931 21813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:54.172114 21813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:54.173643 21813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:54.173693 21813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.173732 21813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:54.173761 21813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.183295 21813 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:38577
I20260928 02:50:54.183326 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:38577 every 8 connection(s)
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:54.184247 21813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:54.187348 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.174455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:55245 (local address 127.19.139.190:35031)
0928 02:50:54.174603 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.174608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.175437 (+ 829us) server_negotiation.cc:407] Connection header received
0928 02:50:54.176610 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.176614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.176669 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.176754 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.178369 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.179178 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.180906 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.181146 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.181237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.183986 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.183990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.183993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.184021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.185547 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.186024 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.186027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.186028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.186108 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.186371 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.186373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.186374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.186600 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.186672 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.187115 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.187224 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:54.187498 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21813
I20260928 02:50:54.187616 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:54.187901 20014 external_mini_cluster.cc:1428] Setting key 208a7e650d3397d14c9cf8e5f1c506b1
I20260928 02:50:54.188473 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.174818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.19.139.129:55245)
0928 02:50:54.175276 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.175326 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.176364 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.176978 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.176987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.177468 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.178194 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.178214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.179306 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.179309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.180777 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.180788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.181307 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.181312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.181541 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.182212 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.182228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.183809 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.185648 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.185652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.185664 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.185935 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.186175 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.186177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.186178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.186308 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.186673 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.186677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.186924 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.187083 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.187394 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260928 02:50:54.189787 21944 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35031
I20260928 02:50:54.189952 21944 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:54.190542 21944 heartbeater.cc:511] Master 127.19.139.190:35031 requested a full tablet report, sending...
I20260928 02:50:54.192291 21740 ts_manager.cc:194] Registered new tserver with Master: a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577)
I20260928 02:50:54.193548 21740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:55245
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:50:54.199662 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.191970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:50045 (local address 127.19.139.190:35031)
0928 02:50:54.192083 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.192087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.192292 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:50:54.192405 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.192408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.192455 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.192539 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.193048 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.193780 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.194831 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.195144 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.195251 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.196553 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.196556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.196558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.196579 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.198068 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.198470 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.198474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.198475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.198555 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.198806 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.198809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.198811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.199006 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.199096 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.199350 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.199500 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:54.199694 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.191878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.19.139.129:50045)
0928 02:50:54.192215 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.192229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.192326 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.192566 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.192569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.192782 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.192915 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.192923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.193924 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.193927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.194711 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.194722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.195269 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.195272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.195377 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.195919 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.195943 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.196426 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.198178 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.198181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.198183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.198380 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.198640 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.198643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.198645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.198727 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.199108 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.199111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.199203 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.199287 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.199445 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:50:54.215679 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35031
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:54.326771 21953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:54.327057 21953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:54.327105 21953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:54.331264 21953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:54.331338 21953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:54.331382 21953 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:50:54.331403 21953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:50:54.331455 21953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:50:54.336921 21953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:54.338039 21953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:54.339075 21953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:54.348575 21961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:54.348701 21953 server_base.cc:1034] running on GCE node
W20260928 02:50:54.348575 21958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:54.348862 21959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:54.349339 21953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:54.349995 21953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:54.351147 21953 hybrid_clock.cc:648] HybridClock initialized: now 1790563854351102 us; error 58 us; skew 500 ppm
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:54.353927 21953 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:50:54.354853 21953 webserver.cc:460] Webserver started at http://127.19.139.130:34109/ using document root <none> and password file <none>
I20260928 02:50:54.355389 21953 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:54.355448 21953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:54.355630 21953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:54.357874 21953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "cde7adf81fe44727b211d51e4d954444"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "b1bb125787cd11bbede061b166116c11"
server_key_iv: "ede446c9ca66c00fe6da3fd5a7d99f14"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.358584 21953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "cde7adf81fe44727b211d51e4d954444"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "b1bb125787cd11bbede061b166116c11"
server_key_iv: "ede446c9ca66c00fe6da3fd5a7d99f14"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.362962 21953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 02:50:54.365741 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.367085 21953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:50:54.367199 21953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "cde7adf81fe44727b211d51e4d954444"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "b1bb125787cd11bbede061b166116c11"
server_key_iv: "ede446c9ca66c00fe6da3fd5a7d99f14"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.367316 21953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:54.389029 21953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:54.392454 21953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:54.392616 21953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:54.394045 21953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:54.394098 21953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.394140 21953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:54.394167 21953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:54.404655 21953 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:37211
I20260928 02:50:54.404702 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:37211 every 8 connection(s)
I20260928 02:50:54.405581 21953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:54.408455 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.394932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:37575 (local address 127.19.139.190:35031)
0928 02:50:54.395047 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.395050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.396186 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 02:50:54.397275 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.397279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.397343 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.397455 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.398798 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.399334 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.401247 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.401598 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.401710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.404766 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.404771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.404774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.404806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.406385 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.406901 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.406904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.406905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.406987 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.407260 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.407262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.407263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.407511 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.407588 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.408095 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.408325 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:50:54.409096 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.395514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.19.139.130:37575)
0928 02:50:54.396013 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.396057 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.396979 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.397647 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.397656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.398076 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.398654 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.398669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.399452 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.399456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.401112 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.401125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.401783 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.401787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.402008 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.402638 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.402661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.404573 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.406511 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.406516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.406528 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.406801 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.407047 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.407049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.407050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.407173 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.407588 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.407592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.407817 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.407981 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.408284 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260928 02:50:54.410384 22084 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35031
I20260928 02:50:54.410534 22084 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:54.411046 22084 heartbeater.cc:511] Master 127.19.139.190:35031 requested a full tablet report, sending...
I20260928 02:50:54.412081 21741 ts_manager.cc:194] Registered new tserver with Master: cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211)
I20260928 02:50:54.412322 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 21953
I20260928 02:50:54.412426 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:54.412698 20014 external_mini_cluster.cc:1428] Setting key 9b91387dade73b91c7ca4b9b4c3b463b
I20260928 02:50:54.412842 21741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:37575
I20260928 02:50:54.420004 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.413065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:48993 (local address 127.19.139.190:35031)
0928 02:50:54.413307 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.413311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.413325 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:54.413360 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.413362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.413401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.413483 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.413951 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.414641 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.415483 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.415688 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.415761 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.416933 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.416937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.416939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.416962 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.418435 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.418819 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.418822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.418823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.418897 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.419203 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.419207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.419208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.419413 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.419471 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.419751 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.419852 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260928 02:50:54.420058 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.412428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.19.139.130:48993)
0928 02:50:54.412701 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.412715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.412794 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.413502 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.413505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.413711 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.413817 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.413825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.414774 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.414777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.415366 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.415377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.415780 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.415784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.415873 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.416412 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.416426 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.416799 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.418529 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.418532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.418534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.418725 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.418979 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.418982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.418984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.419064 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.419530 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.419532 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.419605 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.419683 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.419829 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:50:54.439194 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35031
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:54.550025 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:54.550314 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:54.550369 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:54.554487 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:54.554562 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:54.554605 22093 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:50:54.554617 22093 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:50:54.554656 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:50:54.560258 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:54.561430 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:54.562589 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:54.572647 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:54.572664 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:54.572667 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:54.572839 22093 server_base.cc:1034] running on GCE node
I20260928 02:50:54.573310 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:54.573931 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:54.575093 22093 hybrid_clock.cc:648] HybridClock initialized: now 1790563854575080 us; error 24 us; skew 500 ppm
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:54.577877 22093 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:50:54.578790 22093 webserver.cc:460] Webserver started at http://127.19.139.131:45695/ using document root <none> and password file <none>
I20260928 02:50:54.579427 22093 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:54.579497 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:54.579707 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:54.581880 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "c82c27d0bac347949d292fc0ac6810b1"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "4837fe607bd3bda9db3f0218143d7b6f"
server_key_iv: "b11026833a8158d6178b73f87a91ab31"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.582408 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "c82c27d0bac347949d292fc0ac6810b1"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "4837fe607bd3bda9db3f0218143d7b6f"
server_key_iv: "b11026833a8158d6178b73f87a91ab31"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.586212 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:50:54.588562 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.589782 22093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:54.589875 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "c82c27d0bac347949d292fc0ac6810b1"
format_stamp: "Formatted at 2026-09-28 02:50:54 on dist-test-slave-5vc0"
server_key: "4837fe607bd3bda9db3f0218143d7b6f"
server_key_iv: "b11026833a8158d6178b73f87a91ab31"
server_key_version: "encryptionkey@0"
I20260928 02:50:54.589963 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:54.619752 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:54.622676 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:54.622872 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:54.624660 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:54.624709 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.624753 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:54.624780 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:54.634966 22093 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:40175
I20260928 02:50:54.635000 22223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:40175 every 8 connection(s)
I20260928 02:50:54.636039 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:54.640520 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.627075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:40921 (local address 127.19.139.190:35031)
0928 02:50:54.629108 (+ 2033us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.629113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.629131 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:50:54.630572 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.630576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.630632 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.630703 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.632442 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.632967 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.634344 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.634573 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.634657 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.637344 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.637349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.637351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.637378 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.638511 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.638991 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.638994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.638996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.639079 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.639395 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.639398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.639400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.639588 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.639671 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.640195 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.640322 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1864}
I20260928 02:50:54.641299 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.627596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.19.139.131:40921)
0928 02:50:54.628438 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.628491 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.630197 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.631008 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.631018 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.631531 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.632247 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.632273 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.633089 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.633092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.634219 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.634229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.634711 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.634715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.634909 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.635540 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.635557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.637170 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.638617 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.638622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.638637 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.638888 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.639157 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.639160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.639162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.639294 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.639681 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.639687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.639931 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.640085 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.640434 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":137,"threads_started":1}
I20260928 02:50:54.642485 22224 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35031
I20260928 02:50:54.642627 22224 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:54.643110 22224 heartbeater.cc:511] Master 127.19.139.190:35031 requested a full tablet report, sending...
I20260928 02:50:54.644132 21740 ts_manager.cc:194] Registered new tserver with Master: c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:54.644910 21740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:40921
I20260928 02:50:54.646057 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 22093
I20260928 02:50:54.646173 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:54.646449 20014 external_mini_cluster.cc:1428] Setting key 621dd44a51f99783f11528323e175145
I20260928 02:50:54.648504 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:54.651850 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.644532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.19.139.131:48579)
0928 02:50:54.644818 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.644832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.644937 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.645164 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.645167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.645384 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.645504 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.645511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.646479 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.646482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.647223 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.647234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.647749 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.647752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.647842 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.648384 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.648398 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.648752 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.650487 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.650492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.650494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.650739 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.650991 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.650994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.650995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.651083 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.651385 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.651388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.651481 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.651583 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.651713 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260928 02:50:54.651875 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.644593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:48579 (local address 127.19.139.190:35031)
0928 02:50:54.644690 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.644694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.644906 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:50:54.645020 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.645023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.645065 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.645133 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.645608 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.646344 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.647320 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.647594 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.647692 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.648936 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.648940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.648943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.648969 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.650378 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.650822 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.650824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.650825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.650904 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.651153 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.651155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.651156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.651307 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.651375 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.651637 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.651740 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:54.674700 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.666671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.19.139.190:35031)
0928 02:50:54.666790 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.666794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.666805 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:50:54.666846 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.666849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.666896 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.666967 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.667781 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.668381 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.669213 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.669428 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.669500 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.671957 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.671960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.671963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.671992 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.673331 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.673723 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.673725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.673727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.673814 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.674033 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.674035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.674037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.674213 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.674312 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.674472 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.674579 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:50:54.677767 21740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:54.680831 21740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:54.695272 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.689695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38577 (local address 127.0.0.1:53890)
0928 02:50:54.690289 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.690345 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.690515 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.691290 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.691296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.691318 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:54.691452 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.691458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.692627 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.692631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.693776 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.693785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.694673 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.694677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.694813 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.694851 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.694973 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.695105 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260928 02:50:54.695295 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.689850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.19.139.129:38577)
0928 02:50:54.690260 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.690267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.690663 (+ 396us) server_negotiation.cc:407] Connection header received
0928 02:50:54.690748 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.690754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.691042 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.691191 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:54.691572 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.692527 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.693886 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.694503 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.694674 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.694824 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.695029 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.695138 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":87,"threads_started":1}
I20260928 02:50:54.695987 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.690052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:40175 (local address 127.0.0.1:54974)
0928 02:50:54.691039 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.691052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.691149 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.691637 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.691640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.691656 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:54.691751 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.691758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.693248 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.693252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.694572 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.694582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.695581 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.695585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.695675 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.695698 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.695755 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.695827 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":29,"spinlock_wait_cycles":5760,"thread_start_us":104,"threads_started":1}
I20260928 02:50:54.696026 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.690526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.19.139.131:40175)
0928 02:50:54.690931 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.690943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.691241 (+ 298us) server_negotiation.cc:407] Connection header received
0928 02:50:54.691317 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.691322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.691514 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.691657 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:54.691836 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.693129 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.694703 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.695481 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.695610 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.695752 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.695799 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.695881 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":108,"threads_started":1}
I20260928 02:50:54.696720 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.690464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:37211 (local address 127.0.0.1:34826)
0928 02:50:54.691042 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.691052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.691149 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.691745 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.691748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.691787 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:54.691886 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.691892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.693248 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.693252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.694806 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.694814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.696032 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.696036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.696131 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.696451 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.696514 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.696589 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":408,"spinlock_wait_cycles":5120,"thread_start_us":96,"threads_started":1}
I20260928 02:50:54.697127 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.690861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.19.139.130:37211)
0928 02:50:54.691234 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.691244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.691264 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:50:54.691334 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.691339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.691481 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.691617 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:54.692017 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.693120 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.694900 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.695693 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.695832 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.695970 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.696889 (+ 919us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.696990 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":105,"threads_started":1}
I20260928 02:50:54.698676 22141 tablet_service.cc:1467] Processing CreateTablet for tablet 165af474ff004bf1bf1962cc3a034a16 (DEFAULT_TABLE table=test-table [id=a130022cb87e4963b3301d572f4121d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:54.698671 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 165af474ff004bf1bf1962cc3a034a16 (DEFAULT_TABLE table=test-table [id=a130022cb87e4963b3301d572f4121d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:54.699620 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165af474ff004bf1bf1962cc3a034a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:54.699630 22141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165af474ff004bf1bf1962cc3a034a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:54.703591 22018 tablet_service.cc:1467] Processing CreateTablet for tablet 165af474ff004bf1bf1962cc3a034a16 (DEFAULT_TABLE table=test-table [id=a130022cb87e4963b3301d572f4121d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:54.704758 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165af474ff004bf1bf1962cc3a034a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:54.706187 22247 tablet_bootstrap.cc:492] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Bootstrap starting.
I20260928 02:50:54.707866 22248 tablet_bootstrap.cc:492] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Bootstrap starting.
I20260928 02:50:54.708513 22247 tablet_bootstrap.cc:654] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:54.709236 22247 log.cc:824] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:54.710217 22248 tablet_bootstrap.cc:654] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:54.711290 22247 tablet_bootstrap.cc:492] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: No bootstrap required, opened a new log
I20260928 02:50:54.711277 22248 log.cc:824] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:54.711494 22247 ts_tablet_manager.cc:1397] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:50:54.713480 22248 tablet_bootstrap.cc:492] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: No bootstrap required, opened a new log
I20260928 02:50:54.713670 22248 ts_tablet_manager.cc:1397] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 02:50:54.715474 22247 raft_consensus.cc:348] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.715713 22247 raft_consensus.cc:374] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:54.715776 22247 raft_consensus.cc:729] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Initialized, Role: FOLLOWER
I20260928 02:50:54.716253 22247 consensus_queue.cc:260] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.716545 22251 tablet_bootstrap.cc:492] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Bootstrap starting.
I20260928 02:50:54.716863 21944 heartbeater.cc:503] Master 127.19.139.190:35031 was elected leader, sending a full tablet report...
I20260928 02:50:54.717175 22248 raft_consensus.cc:348] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.717437 22248 raft_consensus.cc:374] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:54.717492 22248 raft_consensus.cc:729] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:54.718014 22248 consensus_queue.cc:260] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.718197 22247 ts_tablet_manager.cc:1428] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:50:54.719048 22251 tablet_bootstrap.cc:654] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:54.719095 22248 ts_tablet_manager.cc:1428] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 02:50:54.719421 22224 heartbeater.cc:503] Master 127.19.139.190:35031 was elected leader, sending a full tablet report...
I20260928 02:50:54.720947 22251 log.cc:824] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:54.723318 22251 tablet_bootstrap.cc:492] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: No bootstrap required, opened a new log
I20260928 02:50:54.723527 22251 ts_tablet_manager.cc:1397] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:50:54.726526 22251 raft_consensus.cc:348] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.726780 22251 raft_consensus.cc:374] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:54.726845 22251 raft_consensus.cc:729] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde7adf81fe44727b211d51e4d954444, State: Initialized, Role: FOLLOWER
I20260928 02:50:54.727267 22251 consensus_queue.cc:260] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.727949 22084 heartbeater.cc:503] Master 127.19.139.190:35031 was elected leader, sending a full tablet report...
I20260928 02:50:54.728346 22251 ts_tablet_manager.cc:1428] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:50:54.791280 22252 raft_consensus.cc:482] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:54.791507 22252 raft_consensus.cc:504] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.792785 22252 leader_election.cc:288] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211), c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:54.796602 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.793131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:57219 (local address 127.19.139.131:40175)
0928 02:50:54.793256 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.793260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.793532 (+ 272us) server_negotiation.cc:407] Connection header received
0928 02:50:54.793682 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.793686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.793742 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.793825 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:54.794075 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.794640 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.795664 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.796073 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.796200 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.796254 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.796420 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.796496 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:50:54.796615 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.793052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:40175 (local address 127.19.139.129:57219)
0928 02:50:54.793459 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.793470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.793582 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.793832 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.793835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.793849 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:54.793964 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.793968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.794736 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.794740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.795546 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.795555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.796211 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.796215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.796305 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.796324 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.796387 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.796460 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":123,"threads_started":1}
I20260928 02:50:54.796726 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.793205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:55325 (local address 127.19.139.130:37211)
0928 02:50:54.793324 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.793329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.793530 (+ 201us) server_negotiation.cc:407] Connection header received
0928 02:50:54.793961 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.793964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.794003 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.794065 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:54.794326 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.794835 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.795785 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.796219 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.796321 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.796373 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.796546 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.796611 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:50:54.796761 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.793027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:37211 (local address 127.19.139.129:55325)
0928 02:50:54.793437 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.793452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.793574 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.794092 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.794095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.794113 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:54.794220 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.794225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.794940 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.794943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.795674 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.795682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.796321 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.796325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.796438 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.796459 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.796522 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.796588 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":129,"threads_started":1}
I20260928 02:50:54.797214 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165af474ff004bf1bf1962cc3a034a16" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82c27d0bac347949d292fc0ac6810b1" is_pre_election: true
I20260928 02:50:54.797508 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165af474ff004bf1bf1962cc3a034a16" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444" is_pre_election: true
I20260928 02:50:54.797544 22178 raft_consensus.cc:2454] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 0.
I20260928 02:50:54.797771 22038 raft_consensus.cc:2454] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 0.
I20260928 02:50:54.798004 21830 leader_election.cc:302] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, c82c27d0bac347949d292fc0ac6810b1; no voters:
I20260928 02:50:54.798302 22252 raft_consensus.cc:2781] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:54.798391 22252 raft_consensus.cc:482] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:54.798442 22252 raft_consensus.cc:3037] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:54.799474 22252 raft_consensus.cc:504] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.799788 22252 leader_election.cc:288] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 election: Requested vote from peers cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211), c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:54.800096 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165af474ff004bf1bf1962cc3a034a16" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444"
I20260928 02:50:54.800120 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165af474ff004bf1bf1962cc3a034a16" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82c27d0bac347949d292fc0ac6810b1"
I20260928 02:50:54.800240 22038 raft_consensus.cc:3037] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:54.800290 22178 raft_consensus.cc:3037] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:54.801281 22178 raft_consensus.cc:2454] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 1.
I20260928 02:50:54.801568 21830 leader_election.cc:302] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, c82c27d0bac347949d292fc0ac6810b1; no voters:
I20260928 02:50:54.801627 22038 raft_consensus.cc:2454] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 1.
I20260928 02:50:54.801730 22252 raft_consensus.cc:2781] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:54.801955 22252 raft_consensus.cc:686] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 LEADER]: Becoming Leader. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Running, Role: LEADER
I20260928 02:50:54.802304 22252 consensus_queue.cc:237] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.805500 21741 catalog_manager.cc:5495] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129). New cstate: current_term: 1 leader_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:50:54 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563853, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:54.838344 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.830128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.19.139.190:35031)
0928 02:50:54.830257 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:54.830316 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:54.830494 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:50:54.830626 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:54.830629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:54.830665 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:54.830724 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:54.831434 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.831953 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.832801 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.833020 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.833103 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.835508 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:54.835512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:54.835514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:54.835535 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:54.836729 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.837183 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.837187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.837189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.837259 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:54.837490 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:54.837493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:54.837495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:54.837669 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:54.837748 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:54.838036 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:50:54.838214 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":38}
I20260928 02:50:54.838344 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:54.830047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35031 (local address 127.0.0.1:37928)
0928 02:50:54.830398 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:54.830410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:54.830527 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:54.830761 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:54.830786 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:54.831213 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:54.831321 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.831329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.832052 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.832055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.832686 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:54.832695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:54.833119 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:54.833123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:54.833201 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:54.833813 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:54.833826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:54.835344 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:54.836827 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.836838 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.836845 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.837096 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.837337 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:54.837340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:54.837342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:54.837416 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:54.837763 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:54.837771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:54.837876 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:50:54.837979 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:50:54.838146 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":128,"threads_started":1}
I20260928 02:50:54.842383 21741 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:37928:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:50:54.843163 21741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:54.847705 21878 tablet_service.cc:1467] Processing CreateTablet for tablet c50bfba06c244e4eaa33f6bf43956d2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47a4ffaa77ae4b2680f213239cc93f17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:50:54.847923 22141 tablet_service.cc:1467] Processing CreateTablet for tablet c50bfba06c244e4eaa33f6bf43956d2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47a4ffaa77ae4b2680f213239cc93f17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:50:54.848582 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50bfba06c244e4eaa33f6bf43956d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:54.848623 22141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50bfba06c244e4eaa33f6bf43956d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:54.851037 22247 tablet_bootstrap.cc:492] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: Bootstrap starting.
I20260928 02:50:54.851416 22018 tablet_service.cc:1467] Processing CreateTablet for tablet c50bfba06c244e4eaa33f6bf43956d2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47a4ffaa77ae4b2680f213239cc93f17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:50:54.851845 22247 tablet_bootstrap.cc:654] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:54.851962 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50bfba06c244e4eaa33f6bf43956d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:54.852705 22248 tablet_bootstrap.cc:492] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: Bootstrap starting.
I20260928 02:50:54.853315 22247 tablet_bootstrap.cc:492] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: No bootstrap required, opened a new log
I20260928 02:50:54.853380 22247 ts_tablet_manager.cc:1397] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:50:54.853694 22248 tablet_bootstrap.cc:654] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:54.853716 22247 raft_consensus.cc:348] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.853794 22247 raft_consensus.cc:374] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:54.853813 22247 raft_consensus.cc:729] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Initialized, Role: FOLLOWER
I20260928 02:50:54.853919 22247 consensus_queue.cc:260] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.854146 22252 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:54.854158 22247 ts_tablet_manager.cc:1428] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:50:54.854288 22252 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:54.854662 22251 tablet_bootstrap.cc:492] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: Bootstrap starting.
I20260928 02:50:54.855602 22251 tablet_bootstrap.cc:654] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:54.855804 22248 tablet_bootstrap.cc:492] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: No bootstrap required, opened a new log
I20260928 02:50:54.855881 22248 ts_tablet_manager.cc:1397] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:50:54.856375 22248 raft_consensus.cc:348] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.856495 22248 raft_consensus.cc:374] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:54.856515 22248 raft_consensus.cc:729] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:54.856606 22248 consensus_queue.cc:260] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.856810 22248 ts_tablet_manager.cc:1428] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:54.856823 22253 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:54.856973 22253 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:54.857533 22251 tablet_bootstrap.cc:492] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: No bootstrap required, opened a new log
I20260928 02:50:54.857625 22251 ts_tablet_manager.cc:1397] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:54.858080 22251 raft_consensus.cc:348] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.858192 22251 raft_consensus.cc:374] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:54.858232 22251 raft_consensus.cc:729] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde7adf81fe44727b211d51e4d954444, State: Initialized, Role: FOLLOWER
I20260928 02:50:54.858341 22251 consensus_queue.cc:260] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:54.858615 22251 ts_tablet_manager.cc:1428] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:50:54.858610 22255 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:54.858770 22255 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:50:54.887104 22225 tablet.cc:2369] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:54.906951 22085 tablet.cc:2369] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:54.935951 21945 tablet.cc:2369] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:55.196995 22253 raft_consensus.cc:482] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:55.197142 22253 raft_consensus.cc:504] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.198227 22253 leader_election.cc:288] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577), cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211)
I20260928 02:50:55.202366 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.199042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:60855 (local address 127.19.139.130:37211)
0928 02:50:55.199163 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.199167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.199264 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:50:55.199393 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.199396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.199454 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.199526 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:55.199753 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.200258 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.201343 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.201832 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.201951 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.202003 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.202182 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.202254 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:50:55.202375 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.198589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:57007 (local address 127.19.139.129:38577)
0928 02:50:55.198836 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.198841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.198858 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:50:55.199032 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.199035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.199087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.199190 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:55.199539 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.200139 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.201318 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.201769 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.201882 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.201939 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.202131 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.202216 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260928 02:50:55.202392 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.198517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38577 (local address 127.19.139.131:57007)
0928 02:50:55.198773 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:55.198787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:55.198913 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:55.199209 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:55.199211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:55.199234 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:55.199405 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.199413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.200287 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.200291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.201189 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.201200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.201891 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.201895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.202009 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.202034 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:50:55.202097 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:50:55.202176 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":24,"spinlock_wait_cycles":21760,"thread_start_us":80,"threads_started":1}
I20260928 02:50:55.202401 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.198883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:37211 (local address 127.19.139.131:60855)
0928 02:50:55.199188 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:55.199200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:55.199298 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:55.199540 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:55.199543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:55.199560 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:55.199660 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.199666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.200360 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.200363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.201211 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.201220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.201962 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.201966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.202058 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.202075 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:50:55.202148 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:50:55.202212 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":26,"thread_start_us":58,"threads_started":1}
I20260928 02:50:55.202992 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50bfba06c244e4eaa33f6bf43956d2a" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444" is_pre_election: true
I20260928 02:50:55.203150 22038 raft_consensus.cc:2454] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 0.
I20260928 02:50:55.203121 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50bfba06c244e4eaa33f6bf43956d2a" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" is_pre_election: true
I20260928 02:50:55.203342 21898 raft_consensus.cc:2454] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 0.
I20260928 02:50:55.203521 22109 leader_election.cc:302] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c82c27d0bac347949d292fc0ac6810b1, cde7adf81fe44727b211d51e4d954444; no voters:
I20260928 02:50:55.203821 22253 raft_consensus.cc:2781] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:55.203878 22253 raft_consensus.cc:482] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:55.203933 22253 raft_consensus.cc:3037] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:55.204953 22253 raft_consensus.cc:504] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.205292 22253 leader_election.cc:288] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 election: Requested vote from peers a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577), cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211)
I20260928 02:50:55.205669 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50bfba06c244e4eaa33f6bf43956d2a" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa"
I20260928 02:50:55.205709 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50bfba06c244e4eaa33f6bf43956d2a" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444"
I20260928 02:50:55.205785 21898 raft_consensus.cc:3037] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:55.205813 22038 raft_consensus.cc:3037] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:55.206683 21898 raft_consensus.cc:2454] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 1.
I20260928 02:50:55.206717 22038 raft_consensus.cc:2454] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 1.
I20260928 02:50:55.206954 22109 leader_election.cc:302] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c82c27d0bac347949d292fc0ac6810b1, cde7adf81fe44727b211d51e4d954444; no voters:
I20260928 02:50:55.207104 22253 raft_consensus.cc:2781] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:55.207279 22253 raft_consensus.cc:686] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [term 1 LEADER]: Becoming Leader. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Running, Role: LEADER
I20260928 02:50:55.207598 22253 consensus_queue.cc:237] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.210070 22272 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: TxnStatusTablet state changed. Reason: New leader c82c27d0bac347949d292fc0ac6810b1. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.210196 22272 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:50:55.210605 22274 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:50:55.210824 22274 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:50:55.211186 21740 catalog_manager.cc:5495] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131). New cstate: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } health_report { overall_health: HEALTHY } } }
I20260928 02:50:55.308399 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.304545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.19.139.131:40175)
0928 02:50:55.304766 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.304769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.304782 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:55.304873 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.304875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.304934 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.305019 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:55.305261 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.305796 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.306934 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.307171 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.307241 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.307560 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.307979 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.308057 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.308207 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.308264 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260928 02:50:55.308421 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.304431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:40175 (local address 127.0.0.1:54982)
0928 02:50:55.304621 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:55.304638 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:55.304748 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:55.305048 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:55.305050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:55.305058 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:50:55.305170 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.305176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.305904 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.305908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.306803 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.306815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.307271 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.307274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.307365 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.307423 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.308072 (+ 649us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.308090 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:50:55.308195 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:50:55.308283 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:50:55.313855 22038 raft_consensus.cc:1261] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Refusing update from remote peer c82c27d0bac347949d292fc0ac6810b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:55.313907 21898 raft_consensus.cc:1261] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Refusing update from remote peer c82c27d0bac347949d292fc0ac6810b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:55.314602 22253 consensus_queue.cc:1035] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:55.315122 22272 consensus_queue.cc:1035] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:55.318467 22252 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: TxnStatusTablet state changed. Reason: New leader c82c27d0bac347949d292fc0ac6810b1. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.318609 22252 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:55.319047 22255 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: TxnStatusTablet state changed. Reason: New leader c82c27d0bac347949d292fc0ac6810b1. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.319180 22255 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:55.322139 22253 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.322278 22253 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:50:55.322638 22253 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.322744 22253 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P c82c27d0bac347949d292fc0ac6810b1: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:50:55.323657 22252 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.323776 22252 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P a4bc8283026d4b30bf0c473c6dbb59fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:55.326763 22255 tablet_replica.cc:439] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } } }
I20260928 02:50:55.326906 22255 tablet_replica.cc:442] T c50bfba06c244e4eaa33f6bf43956d2a P cde7adf81fe44727b211d51e4d954444: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:50:55.329792 22280 mvcc.cc:205] Tried to move back new op lower bound from 7334149551357906944 to 7334149550933307392. Current Snapshot: MvccSnapshot[applied={T|T < 7334149551357906944}]
I20260928 02:50:55.333784 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.330171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.19.139.129:38577)
0928 02:50:55.330289 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.330294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.330309 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:50:55.330380 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.330383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.330449 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.330554 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:55.330773 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.331275 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.332228 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.332460 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.332708 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.332971 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.333388 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.333495 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.333587 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.333644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:50:55.345362 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.341362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:33867 (local address 127.19.139.131:40175)
0928 02:50:55.341507 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.341511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.341598 (+ 87us) server_negotiation.cc:407] Connection header received
0928 02:50:55.341723 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.341725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.341770 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.341873 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:55.342202 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.342790 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.344226 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.344463 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.344556 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.344939 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.345017 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.345069 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.345168 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.345219 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:50:55.345399 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.341197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:40175 (local address 127.19.139.129:33867)
0928 02:50:55.341516 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:55.341532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:55.341640 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:55.341925 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:55.341928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:55.341937 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:50:55.342071 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.342078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.342892 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.342896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.344057 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.344069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.344574 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.344578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.344684 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.344747 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.345088 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.345095 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:50:55.345157 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:50:55.345242 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260928 02:50:55.352677 22038 raft_consensus.cc:1261] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Refusing update from remote peer a4bc8283026d4b30bf0c473c6dbb59fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:55.353003 22178 raft_consensus.cc:1261] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Refusing update from remote peer a4bc8283026d4b30bf0c473c6dbb59fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:55.353096 22252 consensus_queue.cc:1035] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:55.353570 22292 consensus_queue.cc:1035] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:55.356196 22279 mvcc.cc:205] Tried to move back new op lower bound from 7334149551520591872 to 7334149549272760320. Current Snapshot: MvccSnapshot[applied={T|T < 7334149551520591872}]
I20260928 02:50:55.358834 22278 mvcc.cc:205] Tried to move back new op lower bound from 7334149551520591872 to 7334149549272760320. Current Snapshot: MvccSnapshot[applied={T|T < 7334149551520591872}]
I20260928 02:50:55.374461 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.371061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:36081 (local address 127.19.139.129:38577)
0928 02:50:55.371204 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.371208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.371339 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:50:55.371446 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.371448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.371492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.371572 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:55.371807 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.372535 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.373480 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.373705 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.373787 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.374052 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.374127 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.374181 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.374266 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.374321 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:50:55.374483 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.370976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38577 (local address 127.19.139.131:36081)
0928 02:50:55.371277 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:55.371290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:55.371381 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:55.371598 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:55.371601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:55.371609 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:50:55.371716 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.371721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.372736 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.372739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.373364 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.373372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.373797 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.373800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.373889 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.373935 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.374194 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.374201 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:50:55.374257 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:50:55.374349 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":59,"threads_started":1}
I20260928 02:50:55.396111 21740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:55.396370 21740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:55.399057 21740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Sending DeleteTablet for 3 replicas of tablet 165af474ff004bf1bf1962cc3a034a16
I20260928 02:50:55.399972 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 165af474ff004bf1bf1962cc3a034a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:55 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:53890
I20260928 02:50:55.400175 22018 tablet_service.cc:1514] Processing DeleteTablet for tablet 165af474ff004bf1bf1962cc3a034a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:55 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:34826
I20260928 02:50:55.400218 22141 tablet_service.cc:1514] Processing DeleteTablet for tablet 165af474ff004bf1bf1962cc3a034a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:55 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54974
I20260928 02:50:55.400595 22300 tablet_replica.cc:330] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: stopping tablet replica
I20260928 02:50:55.400805 22302 tablet_replica.cc:330] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: stopping tablet replica
I20260928 02:50:55.401059 22300 raft_consensus.cc:2229] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:55.401117 22302 raft_consensus.cc:2229] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:55.401427 22302 raft_consensus.cc:2258] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:55.401459 22300 raft_consensus.cc:2258] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:55.401587 21740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:50:55.401835 22301 tablet_replica.cc:330] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: stopping tablet replica
W20260928 02:50:55.402117 21740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:55.402176 22301 raft_consensus.cc:2229] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:55.402420 22301 raft_consensus.cc:2258] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:55.403426 22302 ts_tablet_manager.cc:1905] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:55.404078 22300 ts_tablet_manager.cc:1905] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:55.407173 22300 ts_tablet_manager.cc:1918] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:55.407272 22300 log.cc:1196] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/165af474ff004bf1bf1962cc3a034a16
I20260928 02:50:55.407569 22300 ts_tablet_manager.cc:1939] T 165af474ff004bf1bf1962cc3a034a16 P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting consensus metadata
I20260928 02:50:55.407850 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 8a984c021cd84f8caf990d29f9880192 (DEFAULT_TABLE table=test-table [id=6f078a4b8704447d816ad5fb1153125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:55.408104 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a984c021cd84f8caf990d29f9880192. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:55.409071 22018 tablet_service.cc:1467] Processing CreateTablet for tablet 8a984c021cd84f8caf990d29f9880192 (DEFAULT_TABLE table=test-table [id=6f078a4b8704447d816ad5fb1153125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:55.409319 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a984c021cd84f8caf990d29f9880192. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:55.409492 22141 tablet_service.cc:1467] Processing CreateTablet for tablet 8a984c021cd84f8caf990d29f9880192 (DEFAULT_TABLE table=test-table [id=6f078a4b8704447d816ad5fb1153125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:55.409762 22141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a984c021cd84f8caf990d29f9880192. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:55.412045 21725 catalog_manager.cc:4841] TS a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577): tablet 165af474ff004bf1bf1962cc3a034a16 (table test-table [id=a130022cb87e4963b3301d572f4121d3]) successfully deleted
I20260928 02:50:55.413371 22301 ts_tablet_manager.cc:1905] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:55.415598 22305 tablet_bootstrap.cc:492] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Bootstrap starting.
I20260928 02:50:55.416110 22301 ts_tablet_manager.cc:1918] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:55.416242 22301 log.cc:1196] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/165af474ff004bf1bf1962cc3a034a16
I20260928 02:50:55.416539 22301 ts_tablet_manager.cc:1939] T 165af474ff004bf1bf1962cc3a034a16 P cde7adf81fe44727b211d51e4d954444: Deleting consensus metadata
I20260928 02:50:55.416646 22305 tablet_bootstrap.cc:654] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:55.417779 21725 catalog_manager.cc:4841] TS cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211): tablet 165af474ff004bf1bf1962cc3a034a16 (table test-table [id=a130022cb87e4963b3301d572f4121d3]) successfully deleted
I20260928 02:50:55.418077 22303 tablet_bootstrap.cc:492] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Bootstrap starting.
I20260928 02:50:55.419006 22303 tablet_bootstrap.cc:654] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:55.419691 22302 ts_tablet_manager.cc:1918] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:55.419795 22302 log.cc:1196] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/165af474ff004bf1bf1962cc3a034a16
I20260928 02:50:55.420104 22302 ts_tablet_manager.cc:1939] T 165af474ff004bf1bf1962cc3a034a16 P c82c27d0bac347949d292fc0ac6810b1: Deleting consensus metadata
I20260928 02:50:55.421396 22304 tablet_bootstrap.cc:492] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Bootstrap starting.
I20260928 02:50:55.421398 21726 catalog_manager.cc:4841] TS c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175): tablet 165af474ff004bf1bf1962cc3a034a16 (table test-table [id=a130022cb87e4963b3301d572f4121d3]) successfully deleted
I20260928 02:50:55.422276 22304 tablet_bootstrap.cc:654] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:55.423565 22305 tablet_bootstrap.cc:492] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: No bootstrap required, opened a new log
I20260928 02:50:55.423699 22305 ts_tablet_manager.cc:1397] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 02:50:55.423758 22303 tablet_bootstrap.cc:492] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: No bootstrap required, opened a new log
I20260928 02:50:55.423864 22303 ts_tablet_manager.cc:1397] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:50:55.424291 22303 raft_consensus.cc:348] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.424418 22303 raft_consensus.cc:374] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:55.424456 22303 raft_consensus.cc:729] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde7adf81fe44727b211d51e4d954444, State: Initialized, Role: FOLLOWER
I20260928 02:50:55.424364 22305 raft_consensus.cc:348] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.424578 22303 consensus_queue.cc:260] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.424798 22305 raft_consensus.cc:374] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:55.424837 22303 ts_tablet_manager.cc:1428] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:55.424876 22305 raft_consensus.cc:729] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Initialized, Role: FOLLOWER
I20260928 02:50:55.425093 22305 consensus_queue.cc:260] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.425474 22304 tablet_bootstrap.cc:492] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: No bootstrap required, opened a new log
I20260928 02:50:55.425563 22304 ts_tablet_manager.cc:1397] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 02:50:55.425973 22304 raft_consensus.cc:348] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.426084 22304 raft_consensus.cc:374] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:55.426111 22304 raft_consensus.cc:729] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:55.426251 22304 consensus_queue.cc:260] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.426429 22305 ts_tablet_manager.cc:1428] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:50:55.426523 22304 ts_tablet_manager.cc:1428] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:55.454507 22272 raft_consensus.cc:482] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:55.454656 22272 raft_consensus.cc:504] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.455077 22272 leader_election.cc:288] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577), cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211)
I20260928 02:50:55.455590 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a984c021cd84f8caf990d29f9880192" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" is_pre_election: true
I20260928 02:50:55.455639 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a984c021cd84f8caf990d29f9880192" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444" is_pre_election: true
I20260928 02:50:55.455752 21898 raft_consensus.cc:2454] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 0.
I20260928 02:50:55.455797 22038 raft_consensus.cc:2454] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 0.
I20260928 02:50:55.456104 22109 leader_election.cc:302] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c82c27d0bac347949d292fc0ac6810b1, cde7adf81fe44727b211d51e4d954444; no voters:
I20260928 02:50:55.456327 22272 raft_consensus.cc:2781] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:55.456403 22272 raft_consensus.cc:482] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:55.456440 22272 raft_consensus.cc:3037] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:55.457700 22272 raft_consensus.cc:504] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.458074 22272 leader_election.cc:288] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 election: Requested vote from peers a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577), cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211)
I20260928 02:50:55.458500 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a984c021cd84f8caf990d29f9880192" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444"
I20260928 02:50:55.458520 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a984c021cd84f8caf990d29f9880192" candidate_uuid: "c82c27d0bac347949d292fc0ac6810b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa"
I20260928 02:50:55.458636 21898 raft_consensus.cc:3037] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:55.458638 22038 raft_consensus.cc:3037] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:55.459625 21898 raft_consensus.cc:2454] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 1.
I20260928 02:50:55.459728 22038 raft_consensus.cc:2454] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c82c27d0bac347949d292fc0ac6810b1 in term 1.
I20260928 02:50:55.459939 22109 leader_election.cc:302] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, c82c27d0bac347949d292fc0ac6810b1; no voters:
I20260928 02:50:55.460089 22272 raft_consensus.cc:2781] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:55.460216 22272 raft_consensus.cc:686] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 1 LEADER]: Becoming Leader. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Running, Role: LEADER
I20260928 02:50:55.460345 22272 consensus_queue.cc:237] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:55.462183 21739 catalog_manager.cc:5495] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131). New cstate: current_term: 1 leader_uuid: "c82c27d0bac347949d292fc0ac6810b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } health_report { overall_health: HEALTHY } } }
I20260928 02:50:55.479187 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.475894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.19.139.131:40175)
0928 02:50:55.476016 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.476019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.476031 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:50:55.476176 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.476179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.476232 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.476293 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:55.476540 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.477031 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.478167 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.478371 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.478470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.478751 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.478829 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.478885 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.478976 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.479027 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:50:55.488623 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.485409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:40175 (local address 127.19.139.131:58851)
0928 02:50:55.485585 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:55.485598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:55.485667 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:55.485867 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:55.485869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:55.485874 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:50:55.485969 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.485974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.486921 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.486923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.487480 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:55.487486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.487999 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.488002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:55.488072 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.488096 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.488370 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.488374 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:50:55.488420 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:50:55.488486 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":1}
I20260928 02:50:55.488648 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:55.485498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:58851 (local address 127.19.139.131:40175)
0928 02:50:55.485614 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:55.485618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:55.485632 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:55.485736 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:55.485739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:55.485785 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:55.485858 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:55.486048 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.486827 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.487566 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:55.487899 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:55.488000 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:55.488208 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:55.488294 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:55.488373 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:55.488433 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:50:55.488487 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":44}
I20260928 02:50:55.494419 21898 raft_consensus.cc:1261] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Refusing update from remote peer c82c27d0bac347949d292fc0ac6810b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:55.494419 22038 raft_consensus.cc:1261] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Refusing update from remote peer c82c27d0bac347949d292fc0ac6810b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:55.494815 22291 consensus_queue.cc:1035] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:55.495088 22291 consensus_queue.cc:1035] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:50:55.510403 22278 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:50:57.073606 21739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:57.073755 21739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:57.076273 21739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Sending DeleteTablet for 3 replicas of tablet 8a984c021cd84f8caf990d29f9880192
I20260928 02:50:57.076882 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a984c021cd84f8caf990d29f9880192 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:57 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:53890
I20260928 02:50:57.076968 22018 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a984c021cd84f8caf990d29f9880192 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:57 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:34826
I20260928 02:50:57.076968 22141 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a984c021cd84f8caf990d29f9880192 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:57 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54974
I20260928 02:50:57.077387 22334 tablet_replica.cc:330] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: stopping tablet replica
I20260928 02:50:57.077620 22335 tablet_replica.cc:330] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: stopping tablet replica
I20260928 02:50:57.077795 22335 raft_consensus.cc:2229] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:57.077970 22334 raft_consensus.cc:2229] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:57.078174 22335 raft_consensus.cc:2258] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:57.078336 22334 raft_consensus.cc:2258] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:57.079111 22335 ts_tablet_manager.cc:1905] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:57.080228 22334 ts_tablet_manager.cc:1905] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:57.081485 22335 ts_tablet_manager.cc:1918] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:50:57.081555 22335 log.cc:1196] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/8a984c021cd84f8caf990d29f9880192
I20260928 02:50:57.081789 22335 ts_tablet_manager.cc:1939] T 8a984c021cd84f8caf990d29f9880192 P cde7adf81fe44727b211d51e4d954444: Deleting consensus metadata
I20260928 02:50:57.082091 22336 tablet_replica.cc:330] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: stopping tablet replica
I20260928 02:50:57.082222 22336 raft_consensus.cc:2229] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:57.082598 22336 raft_consensus.cc:2258] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:57.082903 22334 ts_tablet_manager.cc:1918] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:50:57.082989 22334 log.cc:1196] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/8a984c021cd84f8caf990d29f9880192
I20260928 02:50:57.082947 21725 catalog_manager.cc:4841] TS cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211): tablet 8a984c021cd84f8caf990d29f9880192 (table test-table [id=6f078a4b8704447d816ad5fb1153125b]) successfully deleted
I20260928 02:50:57.083288 22334 ts_tablet_manager.cc:1939] T 8a984c021cd84f8caf990d29f9880192 P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting consensus metadata
I20260928 02:50:57.083506 22336 ts_tablet_manager.cc:1905] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:57.084021 21725 catalog_manager.cc:4841] TS a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577): tablet 8a984c021cd84f8caf990d29f9880192 (table test-table [id=6f078a4b8704447d816ad5fb1153125b]) successfully deleted
I20260928 02:50:57.085951 22336 ts_tablet_manager.cc:1918] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:50:57.086019 22336 log.cc:1196] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/8a984c021cd84f8caf990d29f9880192
I20260928 02:50:57.086242 22336 ts_tablet_manager.cc:1939] T 8a984c021cd84f8caf990d29f9880192 P c82c27d0bac347949d292fc0ac6810b1: Deleting consensus metadata
I20260928 02:50:57.087054 21726 catalog_manager.cc:4841] TS c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175): tablet 8a984c021cd84f8caf990d29f9880192 (table test-table [id=6f078a4b8704447d816ad5fb1153125b]) successfully deleted
Sep 28 02:50:57 dist-test-slave-5vc0 krb5kdc[21696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:50:57 dist-test-slave-5vc0 krb5kdc[21696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:50:57.104059 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:57.095459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.19.139.190:35031)
0928 02:50:57.095705 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:57.095711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:57.095725 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:57.095852 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:57.095854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:57.095899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:57.095991 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:57.096807 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:57.097374 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:57.098220 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:57.098467 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:57.098561 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:57.101019 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:57.101023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:57.101025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:57.101054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:57.102502 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:57.102888 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:57.102891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:57.102892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:57.102977 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:57.103184 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:57.103187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:57.103189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:57.103422 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:57.103529 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:57.103810 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:50:57.103920 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260928 02:50:57.106712 21739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:57.107113 21739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:57.111471 21878 tablet_service.cc:1467] Processing CreateTablet for tablet f4bea9978e064e0a88b0cabe4f9fbc2f (DEFAULT_TABLE table=test-table [id=46b59b60fab44922b71ee1873e3b8744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:57.111488 22018 tablet_service.cc:1467] Processing CreateTablet for tablet f4bea9978e064e0a88b0cabe4f9fbc2f (DEFAULT_TABLE table=test-table [id=46b59b60fab44922b71ee1873e3b8744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:57.111605 22141 tablet_service.cc:1467] Processing CreateTablet for tablet f4bea9978e064e0a88b0cabe4f9fbc2f (DEFAULT_TABLE table=test-table [id=46b59b60fab44922b71ee1873e3b8744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:57.111795 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bea9978e064e0a88b0cabe4f9fbc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:57.111796 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bea9978e064e0a88b0cabe4f9fbc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:57.111836 22141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bea9978e064e0a88b0cabe4f9fbc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:57.114506 22347 tablet_bootstrap.cc:492] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Bootstrap starting.
I20260928 02:50:57.114552 22346 tablet_bootstrap.cc:492] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Bootstrap starting.
I20260928 02:50:57.114276 22345 tablet_bootstrap.cc:492] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Bootstrap starting.
I20260928 02:50:57.115378 22346 tablet_bootstrap.cc:654] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:57.115458 22347 tablet_bootstrap.cc:654] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:57.116281 22345 tablet_bootstrap.cc:654] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:57.116475 22346 tablet_bootstrap.cc:492] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: No bootstrap required, opened a new log
I20260928 02:50:57.116557 22346 ts_tablet_manager.cc:1397] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:57.116964 22346 raft_consensus.cc:348] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.117086 22346 raft_consensus.cc:374] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:57.117118 22346 raft_consensus.cc:729] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:57.117242 22346 consensus_queue.cc:260] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.117491 22345 tablet_bootstrap.cc:492] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: No bootstrap required, opened a new log
I20260928 02:50:57.117501 22346 ts_tablet_manager.cc:1428] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:57.117570 22345 ts_tablet_manager.cc:1397] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:50:57.118013 22345 raft_consensus.cc:348] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.118127 22345 raft_consensus.cc:374] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:57.118151 22345 raft_consensus.cc:729] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde7adf81fe44727b211d51e4d954444, State: Initialized, Role: FOLLOWER
I20260928 02:50:57.118273 22345 consensus_queue.cc:260] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.118662 22345 ts_tablet_manager.cc:1428] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:50:57.119156 22347 tablet_bootstrap.cc:492] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: No bootstrap required, opened a new log
I20260928 02:50:57.119246 22347 ts_tablet_manager.cc:1397] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 02:50:57.119633 22347 raft_consensus.cc:348] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.119725 22347 raft_consensus.cc:374] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:57.119753 22347 raft_consensus.cc:729] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Initialized, Role: FOLLOWER
I20260928 02:50:57.119899 22347 consensus_queue.cc:260] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.120276 22347 ts_tablet_manager.cc:1428] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:57.139933 22352 raft_consensus.cc:482] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:57.140081 22352 raft_consensus.cc:504] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.140582 22352 leader_election.cc:288] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211), c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:57.141028 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bea9978e064e0a88b0cabe4f9fbc2f" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82c27d0bac347949d292fc0ac6810b1" is_pre_election: true
I20260928 02:50:57.141108 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bea9978e064e0a88b0cabe4f9fbc2f" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444" is_pre_election: true
I20260928 02:50:57.141168 22178 raft_consensus.cc:2454] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 0.
I20260928 02:50:57.141271 22038 raft_consensus.cc:2454] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 0.
I20260928 02:50:57.141499 21830 leader_election.cc:302] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, c82c27d0bac347949d292fc0ac6810b1; no voters:
I20260928 02:50:57.141672 22352 raft_consensus.cc:2781] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:57.141736 22352 raft_consensus.cc:482] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:57.141764 22352 raft_consensus.cc:3037] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:57.142942 22352 raft_consensus.cc:504] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.143255 22352 leader_election.cc:288] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 election: Requested vote from peers cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211), c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:57.143633 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bea9978e064e0a88b0cabe4f9fbc2f" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444"
I20260928 02:50:57.143633 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bea9978e064e0a88b0cabe4f9fbc2f" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82c27d0bac347949d292fc0ac6810b1"
I20260928 02:50:57.143754 22038 raft_consensus.cc:3037] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:57.143754 22178 raft_consensus.cc:3037] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:57.144840 22038 raft_consensus.cc:2454] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 1.
I20260928 02:50:57.144840 22178 raft_consensus.cc:2454] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 1.
I20260928 02:50:57.145118 21830 leader_election.cc:302] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, c82c27d0bac347949d292fc0ac6810b1, cde7adf81fe44727b211d51e4d954444; no voters:
I20260928 02:50:57.145282 22352 raft_consensus.cc:2781] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:57.145490 22352 raft_consensus.cc:686] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 LEADER]: Becoming Leader. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Running, Role: LEADER
I20260928 02:50:57.145622 22352 consensus_queue.cc:237] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.147241 21740 catalog_manager.cc:5495] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129). New cstate: current_term: 1 leader_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:57.163158 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:57.160149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.19.139.129:38577)
0928 02:50:57.160436 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:57.160443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:57.160454 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:50:57.160495 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:57.160498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:57.160541 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:57.160610 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:57.160859 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:57.161390 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:57.162287 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:57.162527 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:57.162604 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:57.162789 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:57.162858 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:57.162908 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:57.162982 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:50:57.163030 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260928 02:50:57.169561 22178 raft_consensus.cc:1261] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Refusing update from remote peer a4bc8283026d4b30bf0c473c6dbb59fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:57.169574 22038 raft_consensus.cc:1261] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Refusing update from remote peer a4bc8283026d4b30bf0c473c6dbb59fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:57.169925 22353 consensus_queue.cc:1035] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:57.170081 22352 consensus_queue.cc:1035] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:57.199491 21740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:57.199668 21740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:57.201871 21740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Sending DeleteTablet for 3 replicas of tablet f4bea9978e064e0a88b0cabe4f9fbc2f
I20260928 02:50:57.202589 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bea9978e064e0a88b0cabe4f9fbc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:57 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:53890
I20260928 02:50:57.202595 22018 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bea9978e064e0a88b0cabe4f9fbc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:57 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:34826
I20260928 02:50:57.202587 22141 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bea9978e064e0a88b0cabe4f9fbc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:57 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54974
I20260928 02:50:57.202778 22334 tablet_replica.cc:330] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: stopping tablet replica
I20260928 02:50:57.202780 22336 tablet_replica.cc:330] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: stopping tablet replica
I20260928 02:50:57.202781 22335 tablet_replica.cc:330] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: stopping tablet replica
I20260928 02:50:57.202894 22335 raft_consensus.cc:2229] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:57.202905 22334 raft_consensus.cc:2229] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:57.202909 22336 raft_consensus.cc:2229] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:57.203022 22336 raft_consensus.cc:2258] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:57.203022 22335 raft_consensus.cc:2258] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:57.203068 22334 raft_consensus.cc:2258] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:57.203868 22335 ts_tablet_manager.cc:1905] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:57.203876 22334 ts_tablet_manager.cc:1905] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:57.203896 22336 ts_tablet_manager.cc:1905] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:57.206046 22336 ts_tablet_manager.cc:1918] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:57.206173 22336 log.cc:1196] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/f4bea9978e064e0a88b0cabe4f9fbc2f
I20260928 02:50:57.206359 22334 ts_tablet_manager.cc:1918] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:57.206358 22335 ts_tablet_manager.cc:1918] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:50:57.206399 22336 ts_tablet_manager.cc:1939] T f4bea9978e064e0a88b0cabe4f9fbc2f P c82c27d0bac347949d292fc0ac6810b1: Deleting consensus metadata
I20260928 02:50:57.206427 22335 log.cc:1196] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/f4bea9978e064e0a88b0cabe4f9fbc2f
I20260928 02:50:57.206427 22334 log.cc:1196] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/f4bea9978e064e0a88b0cabe4f9fbc2f
I20260928 02:50:57.206660 22335 ts_tablet_manager.cc:1939] T f4bea9978e064e0a88b0cabe4f9fbc2f P cde7adf81fe44727b211d51e4d954444: Deleting consensus metadata
I20260928 02:50:57.206660 22334 ts_tablet_manager.cc:1939] T f4bea9978e064e0a88b0cabe4f9fbc2f P a4bc8283026d4b30bf0c473c6dbb59fa: Deleting consensus metadata
I20260928 02:50:57.207258 21726 catalog_manager.cc:4841] TS c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175): tablet f4bea9978e064e0a88b0cabe4f9fbc2f (table test-table [id=46b59b60fab44922b71ee1873e3b8744]) successfully deleted
I20260928 02:50:57.207594 21725 catalog_manager.cc:4841] TS cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211): tablet f4bea9978e064e0a88b0cabe4f9fbc2f (table test-table [id=46b59b60fab44922b71ee1873e3b8744]) successfully deleted
I20260928 02:50:57.207877 21725 catalog_manager.cc:4841] TS a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577): tablet f4bea9978e064e0a88b0cabe4f9fbc2f (table test-table [id=46b59b60fab44922b71ee1873e3b8744]) successfully deleted
I20260928 02:50:57.208046 21740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:57.208524 21740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:57.213218 21878 tablet_service.cc:1467] Processing CreateTablet for tablet f5bb5c7e42e44ff98b6448aad7310bed (DEFAULT_TABLE table=test-table [id=a595683e0a46465bb9b7e7a6b9efd747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:57.213227 22018 tablet_service.cc:1467] Processing CreateTablet for tablet f5bb5c7e42e44ff98b6448aad7310bed (DEFAULT_TABLE table=test-table [id=a595683e0a46465bb9b7e7a6b9efd747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:57.213539 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bb5c7e42e44ff98b6448aad7310bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:57.213549 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bb5c7e42e44ff98b6448aad7310bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:57.213498 22141 tablet_service.cc:1467] Processing CreateTablet for tablet f5bb5c7e42e44ff98b6448aad7310bed (DEFAULT_TABLE table=test-table [id=a595683e0a46465bb9b7e7a6b9efd747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:57.213752 22141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bb5c7e42e44ff98b6448aad7310bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:57.215564 22347 tablet_bootstrap.cc:492] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa: Bootstrap starting.
I20260928 02:50:57.215937 22345 tablet_bootstrap.cc:492] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Bootstrap starting.
I20260928 02:50:57.216248 22346 tablet_bootstrap.cc:492] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Bootstrap starting.
I20260928 02:50:57.216475 22347 tablet_bootstrap.cc:654] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:57.216782 22345 tablet_bootstrap.cc:654] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:57.217074 22346 tablet_bootstrap.cc:654] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:57.217702 22347 tablet_bootstrap.cc:492] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa: No bootstrap required, opened a new log
I20260928 02:50:57.217788 22347 ts_tablet_manager.cc:1397] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:57.218057 22346 tablet_bootstrap.cc:492] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: No bootstrap required, opened a new log
I20260928 02:50:57.218137 22346 ts_tablet_manager.cc:1397] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:57.218189 22347 raft_consensus.cc:348] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.218281 22347 raft_consensus.cc:374] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:57.218300 22347 raft_consensus.cc:729] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Initialized, Role: FOLLOWER
I20260928 02:50:57.218402 22347 consensus_queue.cc:260] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.218438 22346 raft_consensus.cc:348] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.218561 22346 raft_consensus.cc:374] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:57.218593 22346 raft_consensus.cc:729] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82c27d0bac347949d292fc0ac6810b1, State: Initialized, Role: FOLLOWER
I20260928 02:50:57.218603 22347 ts_tablet_manager.cc:1428] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:50:57.218724 22346 consensus_queue.cc:260] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.218968 22346 ts_tablet_manager.cc:1428] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:57.219455 22345 tablet_bootstrap.cc:492] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: No bootstrap required, opened a new log
I20260928 02:50:57.219547 22345 ts_tablet_manager.cc:1397] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 02:50:57.219981 22345 raft_consensus.cc:348] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.220101 22345 raft_consensus.cc:374] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:57.220129 22345 raft_consensus.cc:729] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde7adf81fe44727b211d51e4d954444, State: Initialized, Role: FOLLOWER
I20260928 02:50:57.220276 22345 consensus_queue.cc:260] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.220557 22345 ts_tablet_manager.cc:1428] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:50:57.300971 22353 raft_consensus.cc:482] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:57.301106 22353 raft_consensus.cc:504] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.301559 22353 leader_election.cc:288] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211), c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:57.302017 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bb5c7e42e44ff98b6448aad7310bed" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82c27d0bac347949d292fc0ac6810b1" is_pre_election: true
I20260928 02:50:57.302042 22036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bb5c7e42e44ff98b6448aad7310bed" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444" is_pre_election: true
I20260928 02:50:57.302182 22178 raft_consensus.cc:2454] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 0.
I20260928 02:50:57.302182 22036 raft_consensus.cc:2454] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 0.
I20260928 02:50:57.302523 21829 leader_election.cc:302] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, cde7adf81fe44727b211d51e4d954444; no voters:
I20260928 02:50:57.302690 22353 raft_consensus.cc:2781] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:57.302760 22353 raft_consensus.cc:482] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:57.302798 22353 raft_consensus.cc:3037] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:57.303766 22353 raft_consensus.cc:504] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.304076 22353 leader_election.cc:288] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 election: Requested vote from peers cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211), c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175)
I20260928 02:50:57.304464 22036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bb5c7e42e44ff98b6448aad7310bed" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde7adf81fe44727b211d51e4d954444"
I20260928 02:50:57.304461 22178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bb5c7e42e44ff98b6448aad7310bed" candidate_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82c27d0bac347949d292fc0ac6810b1"
I20260928 02:50:57.304577 22178 raft_consensus.cc:3037] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:57.304581 22036 raft_consensus.cc:3037] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:57.305510 22036 raft_consensus.cc:2454] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 1.
I20260928 02:50:57.305534 22178 raft_consensus.cc:2454] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc8283026d4b30bf0c473c6dbb59fa in term 1.
I20260928 02:50:57.305784 21829 leader_election.cc:302] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc8283026d4b30bf0c473c6dbb59fa, cde7adf81fe44727b211d51e4d954444; no voters:
I20260928 02:50:57.305974 22353 raft_consensus.cc:2781] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:57.306088 22353 raft_consensus.cc:686] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 LEADER]: Becoming Leader. State: Replica: a4bc8283026d4b30bf0c473c6dbb59fa, State: Running, Role: LEADER
I20260928 02:50:57.306250 22353 consensus_queue.cc:237] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } }
I20260928 02:50:57.307840 21740 catalog_manager.cc:5495] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa reported cstate change: term changed from 0 to 1, leader changed from <none> to a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129). New cstate: current_term: 1 leader_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bc8283026d4b30bf0c473c6dbb59fa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:57.332587 22178 raft_consensus.cc:1261] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Refusing update from remote peer a4bc8283026d4b30bf0c473c6dbb59fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:57.332944 22036 raft_consensus.cc:1261] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Refusing update from remote peer a4bc8283026d4b30bf0c473c6dbb59fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:57.332971 22353 consensus_queue.cc:1035] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82c27d0bac347949d292fc0ac6810b1" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:57.333328 22353 consensus_queue.cc:1035] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde7adf81fe44727b211d51e4d954444" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:50:57.345342 22318 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:50:58.895102 21741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:58.895265 21741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:58.897948 21741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a81b9f3350774968bf8148e9f3f3f528: Sending DeleteTablet for 3 replicas of tablet f5bb5c7e42e44ff98b6448aad7310bed
I20260928 02:50:58.898625 22018 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bb5c7e42e44ff98b6448aad7310bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:58 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:34826
I20260928 02:50:58.898625 22141 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bb5c7e42e44ff98b6448aad7310bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:58 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54974
I20260928 02:50:58.898829 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bb5c7e42e44ff98b6448aad7310bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:58 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:53890
I20260928 02:50:58.899174 22390 tablet_replica.cc:330] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa: stopping tablet replica
I20260928 02:50:58.899314 22390 raft_consensus.cc:2229] T f5bb5c7e42e44ff98b6448aad7310bed P a4bc8283026d4b30bf0c473c6dbb59fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:58.899485 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21813
I20260928 02:50:58.903051 22389 tablet_replica.cc:330] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: stopping tablet replica
I20260928 02:50:58.903213 22389 raft_consensus.cc:2229] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:58.903349 22389 raft_consensus.cc:2258] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:58.903568 22388 tablet_replica.cc:330] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: stopping tablet replica
I20260928 02:50:58.903752 22388 raft_consensus.cc:2229] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:58.903894 22388 raft_consensus.cc:2258] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:58.904040 22389 ts_tablet_manager.cc:1905] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:58.906555 22389 ts_tablet_manager.cc:1918] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:50:58.906631 22389 log.cc:1196] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/f5bb5c7e42e44ff98b6448aad7310bed
I20260928 02:50:58.906858 22389 ts_tablet_manager.cc:1939] T f5bb5c7e42e44ff98b6448aad7310bed P cde7adf81fe44727b211d51e4d954444: Deleting consensus metadata
I20260928 02:50:58.907948 21725 catalog_manager.cc:4841] TS cde7adf81fe44727b211d51e4d954444 (127.19.139.130:37211): tablet f5bb5c7e42e44ff98b6448aad7310bed (table test-table [id=a595683e0a46465bb9b7e7a6b9efd747]) successfully deleted
I20260928 02:50:58.908710 22388 ts_tablet_manager.cc:1905] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:50:58.908780 21725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:38577 (error 108)
I20260928 02:50:58.908934 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21953
I20260928 02:50:58.910419 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:58.909933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38577 (local address 127.0.0.1:53936)
0928 02:50:58.910229 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:58.910312 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:38577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
W20260928 02:50:58.910559 21725 catalog_manager.cc:4584] TS a4bc8283026d4b30bf0c473c6dbb59fa (127.19.139.129:38577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5bb5c7e42e44ff98b6448aad7310bed: Network error: Client connection negotiation failed: client connection to 127.19.139.129:38577: connect: Connection refused (error 111)
I20260928 02:50:58.911283 22388 ts_tablet_manager.cc:1918] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:50:58.911376 22388 log.cc:1196] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/f5bb5c7e42e44ff98b6448aad7310bed
I20260928 02:50:58.911618 22388 ts_tablet_manager.cc:1939] T f5bb5c7e42e44ff98b6448aad7310bed P c82c27d0bac347949d292fc0ac6810b1: Deleting consensus metadata
I20260928 02:50:58.912433 21726 catalog_manager.cc:4841] TS c82c27d0bac347949d292fc0ac6810b1 (127.19.139.131:40175): tablet f5bb5c7e42e44ff98b6448aad7310bed (table test-table [id=a595683e0a46465bb9b7e7a6b9efd747]) successfully deleted
I20260928 02:50:58.915508 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 22093
I20260928 02:50:58.921089 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 21709
2026-09-28T02:50:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6285 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:58 dist-test-slave-5vc0 krb5kdc[22395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:58 dist-test-slave-5vc0 krb5kdc[22395](info): set up 2 sockets
Sep 28 02:50:58 dist-test-slave-5vc0 krb5kdc[22395](info): commencing operation
krb5kdc: starting...
W20260928 02:50:59.883926 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:50:59 dist-test-slave-5vc0 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:00.007797 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42545
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:42545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:00.118844 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:00.119203 22408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:00.119252 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:00.123469 22408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:00.123548 22408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:00.123569 22408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:00.123584 22408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:00.123597 22408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:00.129218 22408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:42545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:00.130313 22408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:00.131327 22408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:51:00.140327 22408 server_base.cc:1034] running on GCE node
W20260928 02:51:00.140417 22413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.140364 22416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.140417 22414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:00.141712 22408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:00.142865 22408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:00.144053 22408 hybrid_clock.cc:648] HybridClock initialized: now 1790563860144028 us; error 49 us; skew 500 ppm
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:00.147068 22408 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:00.148065 22408 webserver.cc:460] Webserver started at http://127.19.139.190:41025/ using document root <none> and password file <none>
I20260928 02:51:00.148698 22408 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:00.148787 22408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:00.149006 22408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:00.151238 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "43355c0b4f3f41f59fef2192a14718df"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "0170f3645a737df3d3fba0a2ad5992c5"
server_key_iv: "3f7c72ee06b1227758e3f41e39a9e21b"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.151901 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "43355c0b4f3f41f59fef2192a14718df"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "0170f3645a737df3d3fba0a2ad5992c5"
server_key_iv: "3f7c72ee06b1227758e3f41e39a9e21b"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.156198 22408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:51:00.159322 22423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.161012 22408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:00.161167 22408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "43355c0b4f3f41f59fef2192a14718df"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "0170f3645a737df3d3fba0a2ad5992c5"
server_key_iv: "3f7c72ee06b1227758e3f41e39a9e21b"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.161300 22408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:00.180569 22408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:00.183523 22408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:00.183704 22408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:00.190435 22408 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:42545
I20260928 02:51:00.190464 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:42545 every 8 connection(s)
I20260928 02:51:00.191376 22408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:00.194165 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:00.194159 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 22408
I20260928 02:51:00.194322 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:00.194577 20014 external_mini_cluster.cc:1428] Setting key 2b5ad94e705957d9f9d18a888773b8ef
I20260928 02:51:00.201426 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df: Bootstrap starting.
I20260928 02:51:00.204000 22476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df: Neither blocks nor log segments found. Creating new log.
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:00.204882 22476 log.cc:824] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:00.206765 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df: No bootstrap required, opened a new log
I20260928 02:51:00.209651 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.195957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.19.139.190:42545)
0928 02:51:00.196382 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:00.196404 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:00.196447 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:51:00.197544 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:00.197582 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:00.198031 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:00.198414 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:00.199091 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.200640 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.201532 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.202201 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.202357 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.205168 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:00.205173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:00.205192 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:00.205227 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:00.206901 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.207325 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.207329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.207334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.207453 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.207721 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.207724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.207726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.208119 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:00.208332 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:00.208598 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:51:00.208907 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260928 02:51:00.210415 22476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43355c0b4f3f41f59fef2192a14718df" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42545 } }
I20260928 02:51:00.210645 22476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:00.210716 22476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43355c0b4f3f41f59fef2192a14718df, State: Initialized, Role: FOLLOWER
I20260928 02:51:00.211134 22476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43355c0b4f3f41f59fef2192a14718df" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42545 } }
I20260928 02:51:00.211258 22476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:00.211315 22476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:00.211416 22476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:00.212466 22476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43355c0b4f3f41f59fef2192a14718df" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42545 } }
I20260928 02:51:00.212821 22476 leader_election.cc:302] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43355c0b4f3f41f59fef2192a14718df; no voters:
I20260928 02:51:00.213138 22476 leader_election.cc:288] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:00.213270 22481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:00.213521 22481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [term 1 LEADER]: Becoming Leader. State: Replica: 43355c0b4f3f41f59fef2192a14718df, State: Running, Role: LEADER
I20260928 02:51:00.213845 22481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43355c0b4f3f41f59fef2192a14718df" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42545 } }
I20260928 02:51:00.214336 22476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:00.215576 22482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43355c0b4f3f41f59fef2192a14718df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43355c0b4f3f41f59fef2192a14718df" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42545 } } }
I20260928 02:51:00.215621 22483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43355c0b4f3f41f59fef2192a14718df. Latest consensus state: current_term: 1 leader_uuid: "43355c0b4f3f41f59fef2192a14718df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43355c0b4f3f41f59fef2192a14718df" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42545 } } }
I20260928 02:51:00.215798 22482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:00.215817 22483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df [sys.catalog]: This master's current role is: LEADER
W20260928 02:51:00.219111 22496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:51:00.219189 22496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:51:00.219285 22492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:00.219991 22492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:00.225679 22492 catalog_manager.cc:1347] Generated new cluster ID: c140d4b1f9614cbc957957ebbdf4f949
I20260928 02:51:00.225754 22492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:00.248067 22492 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:00.249444 22492 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:00.264034 22492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43355c0b4f3f41f59fef2192a14718df: Generated new TSK 0
I20260928 02:51:00.265020 22492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:00.303882 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42545
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:00.414937 22502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:00.415249 22502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:00.415316 22502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:00.419489 22502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:00.419569 22502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:00.419664 22502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:00.425038 22502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:00.426103 22502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:00.427155 22502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:00.436905 22507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.436919 22508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.436900 22510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:00.437088 22502 server_base.cc:1034] running on GCE node
I20260928 02:51:00.437538 22502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:00.438184 22502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:00.439358 22502 hybrid_clock.cc:648] HybridClock initialized: now 1790563860439332 us; error 44 us; skew 500 ppm
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:00.441998 22502 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:00.442950 22502 webserver.cc:460] Webserver started at http://127.19.139.129:37787/ using document root <none> and password file <none>
I20260928 02:51:00.443490 22502 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:00.443552 22502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:00.443739 22502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:00.445852 22502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "1606d53a99b440c091c1164cfcd63fad"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "3dde605789824954ac9a1a3c488e2a4f"
server_key_iv: "21edc321880c043e65e4ad3a59a9e855"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.446413 22502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "1606d53a99b440c091c1164cfcd63fad"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "3dde605789824954ac9a1a3c488e2a4f"
server_key_iv: "21edc321880c043e65e4ad3a59a9e855"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.450326 22502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:51:00.452898 22517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.454303 22502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:00.454420 22502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "1606d53a99b440c091c1164cfcd63fad"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "3dde605789824954ac9a1a3c488e2a4f"
server_key_iv: "21edc321880c043e65e4ad3a59a9e855"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.454522 22502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:00.475224 22502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:00.478477 22502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:00.478667 22502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:00.479285 22502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:00.480188 22502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:00.480237 22502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.480293 22502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:00.480322 22502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.488951 22502 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:44289
I20260928 02:51:00.488965 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:44289 every 8 connection(s)
I20260928 02:51:00.489926 22502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:00.490640 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 22502
I20260928 02:51:00.490749 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:00.491020 20014 external_mini_cluster.cc:1428] Setting key 17f44a7da3a8637e86b0301662a40065
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:51:00.504549 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.491811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:37125 (local address 127.19.139.190:42545)
0928 02:51:00.491939 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:00.491943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:00.493129 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 02:51:00.494042 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:00.494046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:00.494103 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:00.494175 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:00.495728 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.496314 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.498036 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.498269 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.498376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.501186 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:00.501189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:00.501191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:00.501256 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:00.502559 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.503018 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.503021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.503022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.503100 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.503424 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.503426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.503428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.503636 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:00.503727 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:00.504260 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 02:51:00.504384 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:51:00.505291 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.492279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42545 (local address 127.19.139.129:37125)
0928 02:51:00.492986 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:00.493029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:00.493813 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:00.494386 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:00.494396 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:00.494901 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:00.495578 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:00.495597 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.496462 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.496467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:00.497898 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:00.497911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.498422 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.498426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:00.498632 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.499264 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:00.499286 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:00.501024 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:00.502670 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:00.502675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:00.502687 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:00.502940 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:00.503159 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:00.503163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:00.503164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:00.503339 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:00.503726 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:00.503731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:00.503983 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:51:00.504136 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:51:00.504435 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":113,"threads_started":1}
I20260928 02:51:00.506524 22631 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42545
I20260928 02:51:00.506750 22631 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:00.507217 22631 heartbeater.cc:511] Master 127.19.139.190:42545 requested a full tablet report, sending...
I20260928 02:51:00.508656 22440 ts_manager.cc:194] Registered new tserver with Master: 1606d53a99b440c091c1164cfcd63fad (127.19.139.129:44289)
I20260928 02:51:00.509748 22440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:37125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:00.518998 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42545
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:00.632372 22636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:00.632655 22636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:00.632699 22636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:00.636890 22636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:00.636955 22636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:00.637017 22636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:00.642493 22636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:00.643601 22636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:00.644735 22636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:00.654289 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.654289 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.654289 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:00.654475 22636 server_base.cc:1034] running on GCE node
I20260928 02:51:00.654944 22636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:00.655589 22636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:00.656764 22636 hybrid_clock.cc:648] HybridClock initialized: now 1790563860656741 us; error 34 us; skew 500 ppm
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:00.659524 22636 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:00.660457 22636 webserver.cc:460] Webserver started at http://127.19.139.130:40425/ using document root <none> and password file <none>
I20260928 02:51:00.661002 22636 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:00.661062 22636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:00.661260 22636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:00.663311 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "7c47feaa1a2f40f5a5f4d55bd77d1b80"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "57bc33d6f09b95f1f2048efd83812bbc"
server_key_iv: "406ef7da55d8ddc6541878cb884e1530"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.663861 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "7c47feaa1a2f40f5a5f4d55bd77d1b80"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "57bc33d6f09b95f1f2048efd83812bbc"
server_key_iv: "406ef7da55d8ddc6541878cb884e1530"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.668006 22636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:00.670670 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.672099 22636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:51:00.672338 22636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "7c47feaa1a2f40f5a5f4d55bd77d1b80"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "57bc33d6f09b95f1f2048efd83812bbc"
server_key_iv: "406ef7da55d8ddc6541878cb884e1530"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.672467 22636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:00.694484 22636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:00.697845 22636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:00.698055 22636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:00.698690 22636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:00.699569 22636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:00.699617 22636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.699678 22636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:00.699707 22636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.708359 22636 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:42383
I20260928 02:51:00.708416 22764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:42383 every 8 connection(s)
I20260928 02:51:00.709375 22636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:00.715791 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 22636
I20260928 02:51:00.715900 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:00.716219 20014 external_mini_cluster.cc:1428] Setting key 7d9619fcdab1bfdbd82ea4d7a9ab0196
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:00.723922 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.711121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:48497 (local address 127.19.139.190:42545)
0928 02:51:00.711287 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:00.711292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:00.712409 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 02:51:00.713284 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:00.713287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:00.713336 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:00.713415 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:00.714669 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.715343 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.716819 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.717038 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.717120 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.719968 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:00.719972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:00.719974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:00.720003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:00.721502 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.722154 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.722157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.722158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.722236 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.722657 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.722660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.722661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.722884 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:00.722970 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:00.723661 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 02:51:00.723795 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:51:00.725170 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.711605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42545 (local address 127.19.139.130:48497)
0928 02:51:00.712268 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:00.712311 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:00.713071 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:00.713593 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:00.713599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:00.713945 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:00.714502 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:00.714517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.715460 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.715463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:00.716700 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:00.716711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.717187 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.717191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:00.717376 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.717978 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:00.717994 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:00.719759 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:00.721634 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:00.721642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:00.721662 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:00.722051 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:00.722323 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:00.722327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:00.722329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:00.722533 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:00.722993 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:00.723000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:00.723364 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:51:00.723590 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 02:51:00.723986 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:51:00.726608 22765 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42545
I20260928 02:51:00.726871 22765 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:00.727382 22765 heartbeater.cc:511] Master 127.19.139.190:42545 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:51:00.728533 22440 ts_manager.cc:194] Registered new tserver with Master: 7c47feaa1a2f40f5a5f4d55bd77d1b80 (127.19.139.130:42383)
I20260928 02:51:00.729110 22440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:48497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:00.743332 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42545
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:00.852636 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:00.852897 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:00.852945 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:00.857060 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:00.857131 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:00.857191 22770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:00.862637 22770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:00.863699 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:00.864774 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:00.874344 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.874326 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:00.874344 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:00.874531 22770 server_base.cc:1034] running on GCE node
I20260928 02:51:00.875082 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:00.875771 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:00.876931 22770 hybrid_clock.cc:648] HybridClock initialized: now 1790563860876901 us; error 42 us; skew 500 ppm
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:00.879648 22770 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:00.880625 22770 webserver.cc:460] Webserver started at http://127.19.139.131:37273/ using document root <none> and password file <none>
I20260928 02:51:00.881166 22770 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:00.881234 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:00.881443 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:00.883494 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "b53b55aaade04333b590e79437d167bc"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "cecd171baa93f060d6578ff162c7410a"
server_key_iv: "51088a7e410c8af6e5ee4ae67a728335"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.884033 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "b53b55aaade04333b590e79437d167bc"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "cecd171baa93f060d6578ff162c7410a"
server_key_iv: "51088a7e410c8af6e5ee4ae67a728335"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.887806 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:51:00.890345 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.891582 22770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:00.891697 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "b53b55aaade04333b590e79437d167bc"
format_stamp: "Formatted at 2026-09-28 02:51:00 on dist-test-slave-5vc0"
server_key: "cecd171baa93f060d6578ff162c7410a"
server_key_iv: "51088a7e410c8af6e5ee4ae67a728335"
server_key_version: "encryptionkey@0"
I20260928 02:51:00.891796 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:00.908409 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:00.912038 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:00.912257 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:00.912854 22770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:00.913758 22770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:00.913810 22770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.913856 22770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:00.913883 22770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:00.923178 22770 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:43111
I20260928 02:51:00.923202 22898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:43111 every 8 connection(s)
I20260928 02:51:00.924206 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:00.929771 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 22770
I20260928 02:51:00.929895 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:00.930202 20014 external_mini_cluster.cc:1428] Setting key e4e73d3180b9da4afc7da5db48ed6b20
Sep 28 02:51:00 dist-test-slave-5vc0 krb5kdc[22395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563860, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:00.937533 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.925971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:52295 (local address 127.19.139.190:42545)
0928 02:51:00.926099 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:00.926102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:00.927127 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 02:51:00.928026 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:00.928030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:00.928089 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:00.928184 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:00.929409 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.930137 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.931459 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.931706 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.931820 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.934251 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:00.934254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:00.934256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:00.934281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:00.935514 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.935966 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.935969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.935970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.936049 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:00.936410 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:00.936414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:00.936415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:00.936627 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:00.936724 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:00.937267 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 02:51:00.937397 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:51:00.938237 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:00.926302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42545 (local address 127.19.139.131:52295)
0928 02:51:00.926964 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:00.927007 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:00.927816 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:00.928385 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:00.928393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:00.928753 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:00.929265 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:00.929280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.930267 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.930272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:00.931319 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:00.931329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:00.931853 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:00.931855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:00.932001 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:00.932628 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:00.932645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:00.934057 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:00.935610 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:00.935615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:00.935627 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:00.935867 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:00.936109 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:00.936111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:00.936112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:00.936294 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:00.936723 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:00.936728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:00.936965 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 02:51:00.937114 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 02:51:00.937367 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":115,"threads_started":1}
I20260928 02:51:00.939415 22899 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42545
I20260928 02:51:00.939641 22899 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:00.940104 22899 heartbeater.cc:511] Master 127.19.139.190:42545 requested a full tablet report, sending...
I20260928 02:51:00.941279 22440 ts_manager.cc:194] Registered new tserver with Master: b53b55aaade04333b590e79437d167bc (127.19.139.131:43111)
I20260928 02:51:00.941874 22440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:52295
I20260928 02:51:00.944135 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:51:00.956774 22479 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:00.955264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.19.139.190:42545)
0928 02:51:00.955410 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:00.955414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:00.955554 (+ 140us) server_negotiation.cc:407] Connection header received
0928 02:51:00.955707 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:00.955710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:00.955752 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:00.955816 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:00.956645 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54622: BlockingRecv error: recv got EOF from 127.0.0.1:54622 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:51:00.957957 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 22502
I20260928 02:51:00.962286 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 22636
I20260928 02:51:00.966472 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 22770
I20260928 02:51:00.970857 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 22408
2026-09-28T02:51:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2048 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:01 dist-test-slave-5vc0 krb5kdc[22911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:01 dist-test-slave-5vc0 krb5kdc[22911](info): set up 2 sockets
Sep 28 02:51:01 dist-test-slave-5vc0 krb5kdc[22911](info): commencing operation
krb5kdc: starting...
W20260928 02:51:01.924443 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:02 dist-test-slave-5vc0 krb5kdc[22911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:02.045481 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:38027
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:38027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:02.154706 22924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:02.154991 22924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:02.155030 22924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:02.159125 22924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:02.159194 22924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:02.159214 22924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:02.159228 22924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:02.159242 22924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:02.164611 22924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:38027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:38027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:02.165704 22924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:02.166713 22924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:02.175310 22929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:02.175304 22930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:02.175302 22932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:02.175462 22924 server_base.cc:1034] running on GCE node
I20260928 02:51:02.176357 22924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:02.177523 22924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:02.178685 22924 hybrid_clock.cc:648] HybridClock initialized: now 1790563862178662 us; error 48 us; skew 500 ppm
Sep 28 02:51:02 dist-test-slave-5vc0 krb5kdc[22911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563862, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:02.181651 22924 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:02.182578 22924 webserver.cc:460] Webserver started at http://127.19.139.190:34475/ using document root <none> and password file <none>
I20260928 02:51:02.183141 22924 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:02.183209 22924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:02.183400 22924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:02.185583 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "93bccb26ac83461c9d3bb45a355907ed"
format_stamp: "Formatted at 2026-09-28 02:51:02 on dist-test-slave-5vc0"
server_key: "f4411fcda5f0b0d6e5a2b01430b1b1e6"
server_key_iv: "de9f608638f7c0a147e8a3b4062079f9"
server_key_version: "encryptionkey@0"
I20260928 02:51:02.186172 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "93bccb26ac83461c9d3bb45a355907ed"
format_stamp: "Formatted at 2026-09-28 02:51:02 on dist-test-slave-5vc0"
server_key: "f4411fcda5f0b0d6e5a2b01430b1b1e6"
server_key_iv: "de9f608638f7c0a147e8a3b4062079f9"
server_key_version: "encryptionkey@0"
I20260928 02:51:02.190021 22924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:51:02.192636 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:02.193959 22924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:02.194085 22924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "93bccb26ac83461c9d3bb45a355907ed"
format_stamp: "Formatted at 2026-09-28 02:51:02 on dist-test-slave-5vc0"
server_key: "f4411fcda5f0b0d6e5a2b01430b1b1e6"
server_key_iv: "de9f608638f7c0a147e8a3b4062079f9"
server_key_version: "encryptionkey@0"
I20260928 02:51:02.194185 22924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:02.210320 22924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:02.213708 22924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:02.213928 22924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:02.220930 22924 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:38027
I20260928 02:51:02.221176 22991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:38027 every 8 connection(s)
I20260928 02:51:02.221907 22924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:02.224843 22992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:02.230346 22992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed: Bootstrap starting.
I20260928 02:51:02.231899 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 22924
I20260928 02:51:02.231989 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:02.232265 20014 external_mini_cluster.cc:1428] Setting key de6b35e78fda9afccf889a3e1a9b9bcc
I20260928 02:51:02.232959 22992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:02.233744 22992 log.cc:824] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:02.235814 22992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed: No bootstrap required, opened a new log
I20260928 02:51:02.239964 22992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bccb26ac83461c9d3bb45a355907ed" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38027 } }
I20260928 02:51:02.240293 22992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:02.240352 22992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93bccb26ac83461c9d3bb45a355907ed, State: Initialized, Role: FOLLOWER
I20260928 02:51:02.240849 22992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bccb26ac83461c9d3bb45a355907ed" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38027 } }
I20260928 02:51:02.240979 22992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 28 02:51:02 dist-test-slave-5vc0 krb5kdc[22911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:02.241039 22992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:02.241138 22992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:02.242113 22992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bccb26ac83461c9d3bb45a355907ed" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38027 } }
I20260928 02:51:02.242484 22992 leader_election.cc:302] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93bccb26ac83461c9d3bb45a355907ed; no voters:
I20260928 02:51:02.242848 22992 leader_election.cc:288] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:02.242942 22997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:02.243106 22997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [term 1 LEADER]: Becoming Leader. State: Replica: 93bccb26ac83461c9d3bb45a355907ed, State: Running, Role: LEADER
I20260928 02:51:02.243481 22997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bccb26ac83461c9d3bb45a355907ed" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38027 } }
I20260928 02:51:02.244257 22992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:02.245198 22999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93bccb26ac83461c9d3bb45a355907ed. Latest consensus state: current_term: 1 leader_uuid: "93bccb26ac83461c9d3bb45a355907ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bccb26ac83461c9d3bb45a355907ed" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38027 } } }
I20260928 02:51:02.245342 22999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:02.245654 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93bccb26ac83461c9d3bb45a355907ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bccb26ac83461c9d3bb45a355907ed" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38027 } } }
I20260928 02:51:02.245743 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:02.246205 23006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:02.247623 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:02.233284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.19.139.190:38027)
0928 02:51:02.233711 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:02.233732 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:02.233769 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:51:02.234539 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:02.234559 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:02.234888 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:02.235167 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:02.236077 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:02.237088 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:02.238283 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:02.238747 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:02.238860 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:02.241594 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:02.241600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:02.241614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:02.241641 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:02.243640 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:02.244106 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:02.244112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:02.244118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:02.244287 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:02.244584 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:02.244587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:02.244588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:02.245144 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:02.246083 (+ 939us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:02.246238 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:51:02.246801 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260928 02:51:02.249579 23006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:02.250362 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:51:02.256206 23006 catalog_manager.cc:1347] Generated new cluster ID: 5402f6b6e09e43b5bd53f97411464806
I20260928 02:51:02.256294 23006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:02.263939 23006 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:02.265216 23006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:02.268393 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:02.263622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.19.139.190:38027)
0928 02:51:02.263737 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:02.263741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:02.264203 (+ 462us) server_negotiation.cc:407] Connection header received
0928 02:51:02.264342 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:02.264345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:02.264391 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:02.264456 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:02.264980 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:02.265545 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:02.266365 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:02.266594 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:02.266678 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:02.267474 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:02.267478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:02.267481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:02.267912 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:02.267941 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:02.267958 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:02.268033 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:02.268132 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:51:02.268250 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:51:02.273763 23006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93bccb26ac83461c9d3bb45a355907ed: Generated new TSK 0
I20260928 02:51:02.274676 23006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:02.281733 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 22924
2026-09-28T02:51:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1310 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:02 dist-test-slave-5vc0 krb5kdc[23025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:02 dist-test-slave-5vc0 krb5kdc[23025](info): set up 2 sockets
Sep 28 02:51:02 dist-test-slave-5vc0 krb5kdc[23025](info): commencing operation
krb5kdc: starting...
W20260928 02:51:03.251925 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:03 dist-test-slave-5vc0 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:03.372332 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:38863
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:38863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:03.481582 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:03.481925 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:03.481966 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:03.486160 23038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:03.486244 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:03.486264 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:03.486279 23038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:03.486294 23038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:03.491990 23038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:38863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:03.493150 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:03.494254 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:03.503008 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:03.503012 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:03.503091 23038 server_base.cc:1034] running on GCE node
W20260928 02:51:03.503012 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:03.503849 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:03.505052 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:03.506295 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790563863506267 us; error 41 us; skew 500 ppm
Sep 28 02:51:03 dist-test-slave-5vc0 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563863, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:03.509132 23038 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:03.510100 23038 webserver.cc:460] Webserver started at http://127.19.139.190:46657/ using document root <none> and password file <none>
I20260928 02:51:03.510668 23038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:03.510735 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:03.510939 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:03.513104 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "bc08450f41814587abbca5ad0b2e42da"
format_stamp: "Formatted at 2026-09-28 02:51:03 on dist-test-slave-5vc0"
server_key: "9903cb321ce3394b8d4e97ec317a2641"
server_key_iv: "4aa1f1ff69494ed327ddabb66098fc63"
server_key_version: "encryptionkey@0"
I20260928 02:51:03.513741 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "bc08450f41814587abbca5ad0b2e42da"
format_stamp: "Formatted at 2026-09-28 02:51:03 on dist-test-slave-5vc0"
server_key: "9903cb321ce3394b8d4e97ec317a2641"
server_key_iv: "4aa1f1ff69494ed327ddabb66098fc63"
server_key_version: "encryptionkey@0"
I20260928 02:51:03.517858 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:51:03.520344 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:03.521564 23038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:03.521677 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "bc08450f41814587abbca5ad0b2e42da"
format_stamp: "Formatted at 2026-09-28 02:51:03 on dist-test-slave-5vc0"
server_key: "9903cb321ce3394b8d4e97ec317a2641"
server_key_iv: "4aa1f1ff69494ed327ddabb66098fc63"
server_key_version: "encryptionkey@0"
I20260928 02:51:03.521776 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:03.543239 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:03.546558 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:03.546765 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:03.554085 23038 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:38863
I20260928 02:51:03.554325 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:38863 every 8 connection(s)
I20260928 02:51:03.555094 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:03.558068 23106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:03.558626 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23038
I20260928 02:51:03.558738 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:03.559008 20014 external_mini_cluster.cc:1428] Setting key b329e11836c91361a764bdc61b500c6b
I20260928 02:51:03.565491 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da: Bootstrap starting.
Sep 28 02:51:03 dist-test-slave-5vc0 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:03.567962 23106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:03.568884 23106 log.cc:824] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:03.571239 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da: No bootstrap required, opened a new log
I20260928 02:51:03.573310 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:03.560323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.19.139.190:38863)
0928 02:51:03.560743 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:03.560758 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:03.560798 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:51:03.561665 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:03.561688 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:03.562046 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:03.562454 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:03.563222 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:03.564253 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:03.565102 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:03.565589 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:03.565734 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:03.568427 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:03.568434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:03.568451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:03.568486 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:03.570455 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:03.570912 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:03.570919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:03.570925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:03.571067 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:03.571394 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:03.571398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:03.571399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:03.571816 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:03.572002 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:03.572291 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:51:03.572578 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260928 02:51:03.575693 23106 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc08450f41814587abbca5ad0b2e42da" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38863 } }
I20260928 02:51:03.575925 23106 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:03.575997 23106 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc08450f41814587abbca5ad0b2e42da, State: Initialized, Role: FOLLOWER
I20260928 02:51:03.576426 23106 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc08450f41814587abbca5ad0b2e42da" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38863 } }
I20260928 02:51:03.576555 23106 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:03.576617 23106 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:03.576709 23106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:03.577656 23106 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc08450f41814587abbca5ad0b2e42da" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38863 } }
I20260928 02:51:03.577984 23106 leader_election.cc:302] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc08450f41814587abbca5ad0b2e42da; no voters:
I20260928 02:51:03.578320 23106 leader_election.cc:288] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:03.578536 23111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:03.578714 23111 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [term 1 LEADER]: Becoming Leader. State: Replica: bc08450f41814587abbca5ad0b2e42da, State: Running, Role: LEADER
I20260928 02:51:03.579012 23111 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc08450f41814587abbca5ad0b2e42da" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38863 } }
I20260928 02:51:03.579464 23106 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:03.580578 23112 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc08450f41814587abbca5ad0b2e42da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc08450f41814587abbca5ad0b2e42da" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38863 } } }
I20260928 02:51:03.580708 23112 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:03.580996 23113 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc08450f41814587abbca5ad0b2e42da. Latest consensus state: current_term: 1 leader_uuid: "bc08450f41814587abbca5ad0b2e42da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc08450f41814587abbca5ad0b2e42da" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38863 } } }
I20260928 02:51:03.581079 23113 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da [sys.catalog]: This master's current role is: LEADER
W20260928 02:51:03.584062 23126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:51:03.584131 23126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:51:03.584250 23120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:03.585034 23120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:03.590633 23120 catalog_manager.cc:1347] Generated new cluster ID: a7b9360901a34c20bf4a55da5b4e47ec
I20260928 02:51:03.590706 23120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:03.607123 23120 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:03.608098 23120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:03.624182 23120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc08450f41814587abbca5ad0b2e42da: Generated new TSK 0
I20260928 02:51:03.625003 23120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:03.655712 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38863
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:03.770119 23132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:03.770416 23132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:03.770459 23132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:03.774610 23132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:03.774678 23132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:03.774744 23132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:03.780316 23132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:03.781507 23132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:03.782567 23132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:03.791993 23138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:03.792032 23140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:03.792022 23137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:03.792089 23132 server_base.cc:1034] running on GCE node
I20260928 02:51:03.792611 23132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:03.793233 23132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:03.794392 23132 hybrid_clock.cc:648] HybridClock initialized: now 1790563863794373 us; error 30 us; skew 500 ppm
Sep 28 02:51:03 dist-test-slave-5vc0 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563863, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:03.797073 23132 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:03.797928 23132 webserver.cc:460] Webserver started at http://127.19.139.129:44979/ using document root <none> and password file <none>
I20260928 02:51:03.798454 23132 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:03.798521 23132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:03.798703 23132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:03.800876 23132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "d62307edfc7b439c8cc0bf23d1e3721f"
format_stamp: "Formatted at 2026-09-28 02:51:03 on dist-test-slave-5vc0"
server_key: "c47a0c4daa58aba94f81148646084e08"
server_key_iv: "e2ee62d3bd780a2973b7ac11e3e32e68"
server_key_version: "encryptionkey@0"
I20260928 02:51:03.801426 23132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "d62307edfc7b439c8cc0bf23d1e3721f"
format_stamp: "Formatted at 2026-09-28 02:51:03 on dist-test-slave-5vc0"
server_key: "c47a0c4daa58aba94f81148646084e08"
server_key_iv: "e2ee62d3bd780a2973b7ac11e3e32e68"
server_key_version: "encryptionkey@0"
I20260928 02:51:03.805366 23132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:03.808012 23147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:03.809474 23132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:03.809623 23132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "d62307edfc7b439c8cc0bf23d1e3721f"
format_stamp: "Formatted at 2026-09-28 02:51:03 on dist-test-slave-5vc0"
server_key: "c47a0c4daa58aba94f81148646084e08"
server_key_iv: "e2ee62d3bd780a2973b7ac11e3e32e68"
server_key_version: "encryptionkey@0"
I20260928 02:51:03.809738 23132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:03.827289 23132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:03.830524 23132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:03.830747 23132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:03.831362 23132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:03.832238 23132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:03.832284 23132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:03.832336 23132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:03.832362 23132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:03.841107 23132 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:39549
I20260928 02:51:03.841132 23260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:39549 every 8 connection(s)
I20260928 02:51:03.842077 23132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:03.852085 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23132
I20260928 02:51:03.852301 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
Sep 28 02:51:03 dist-test-slave-5vc0 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563863, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:03.852628 20014 external_mini_cluster.cc:1428] Setting key ee5026678072818365ab3eac6c226422
I20260928 02:51:03.856930 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:03.844018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:40557 (local address 127.19.139.190:38863)
0928 02:51:03.844231 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:03.844235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:03.845126 (+ 891us) server_negotiation.cc:407] Connection header received
0928 02:51:03.846504 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:03.846507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:03.846562 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:03.846650 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:03.847991 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:03.848632 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:03.849984 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:03.850262 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:03.850357 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:03.853145 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:03.853151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:03.853154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:03.853187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:03.854562 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:03.855210 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:03.855213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:03.855214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:03.855309 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:03.855689 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:03.855692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:03.855694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:03.855926 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:03.856035 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:03.856612 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 02:51:03.856754 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:51:03.857721 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:03.844466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:38863 (local address 127.19.139.129:40557)
0928 02:51:03.844959 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:03.845003 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:03.846257 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:03.846819 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:03.846826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:03.847224 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:03.847796 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:03.847811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:03.848825 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:03.848828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:03.849833 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:03.849845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:03.850430 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:03.850433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:03.850601 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:03.851186 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:03.851203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:03.852943 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:03.854687 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:03.854694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:03.854714 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:03.855091 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:03.855400 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:03.855403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:03.855405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:03.855604 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:03.856039 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:03.856044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:03.856372 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 02:51:03.856549 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 02:51:03.856849 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":144,"threads_started":1}
I20260928 02:51:03.858991 23261 heartbeater.cc:347] Connected to a master server at 127.19.139.190:38863
I20260928 02:51:03.859287 23261 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:03.859946 23261 heartbeater.cc:511] Master 127.19.139.190:38863 requested a full tablet report, sending...
I20260928 02:51:03.861627 23070 ts_manager.cc:194] Registered new tserver with Master: d62307edfc7b439c8cc0bf23d1e3721f (127.19.139.129:39549)
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:03.862808 23070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:40557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:03.879842 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38863
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:03.989694 23266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:03.989980 23266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:03.990025 23266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:03.994176 23266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:03.994249 23266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:03.994315 23266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:03.999781 23266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:04.000922 23266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:04.001940 23266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:04.011951 23272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:04.011965 23274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:04.011968 23271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:04.012521 23266 server_base.cc:1034] running on GCE node
I20260928 02:51:04.012965 23266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:04.013652 23266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:04.014823 23266 hybrid_clock.cc:648] HybridClock initialized: now 1790563864014804 us; error 35 us; skew 500 ppm
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563864, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:04.021303 23266 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:04.022361 23266 webserver.cc:460] Webserver started at http://127.19.139.130:37893/ using document root <none> and password file <none>
I20260928 02:51:04.022969 23266 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:04.023048 23266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:04.023260 23266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:04.025520 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "8e7d53fd01fa4257b781dbeacf12f1bf"
format_stamp: "Formatted at 2026-09-28 02:51:04 on dist-test-slave-5vc0"
server_key: "e8f9fa20b3cd59f48bab5e78d30312cd"
server_key_iv: "175c5eb7e5bf749ea656855e39cfeb42"
server_key_version: "encryptionkey@0"
I20260928 02:51:04.026114 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "8e7d53fd01fa4257b781dbeacf12f1bf"
format_stamp: "Formatted at 2026-09-28 02:51:04 on dist-test-slave-5vc0"
server_key: "e8f9fa20b3cd59f48bab5e78d30312cd"
server_key_iv: "175c5eb7e5bf749ea656855e39cfeb42"
server_key_version: "encryptionkey@0"
I20260928 02:51:04.029954 23266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:04.032550 23281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:04.033926 23266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:51:04.034041 23266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "8e7d53fd01fa4257b781dbeacf12f1bf"
format_stamp: "Formatted at 2026-09-28 02:51:04 on dist-test-slave-5vc0"
server_key: "e8f9fa20b3cd59f48bab5e78d30312cd"
server_key_iv: "175c5eb7e5bf749ea656855e39cfeb42"
server_key_version: "encryptionkey@0"
I20260928 02:51:04.034147 23266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:04.050856 23266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:04.053972 23266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:04.054152 23266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:04.054744 23266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:04.055601 23266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:04.055649 23266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:04.055699 23266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:04.055727 23266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:04.064419 23266 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:43695
I20260928 02:51:04.064453 23394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:43695 every 8 connection(s)
I20260928 02:51:04.065305 23266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:04.066354 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23266
I20260928 02:51:04.066474 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:04.066737 20014 external_mini_cluster.cc:1428] Setting key c2d3d00a99e773dea1817452f92938e7
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563864, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:51:04.079218 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:04.067079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:41515 (local address 127.19.139.190:38863)
0928 02:51:04.067199 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:04.067203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:04.068141 (+ 938us) server_negotiation.cc:407] Connection header received
0928 02:51:04.069583 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:04.069586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:04.069644 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:04.069716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:04.070974 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.071501 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.072725 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.072958 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.073049 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:04.075534 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:04.075537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:04.075539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:04.075566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:04.077106 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:04.077650 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:04.077653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:04.077654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:04.077736 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:04.078130 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:04.078133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:04.078135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:04.078330 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:04.078416 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:04.078919 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 02:51:04.079053 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:51:04.080294 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:04.067522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:38863 (local address 127.19.139.130:41515)
0928 02:51:04.067996 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:04.068052 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:04.069350 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:04.069924 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:04.069932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:04.070305 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:04.070812 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:04.070826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.071629 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.071632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:04.072598 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:04.072608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.073110 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.073115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:04.073286 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:04.073910 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:04.073926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:04.075362 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:04.077228 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:04.077234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:04.077252 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:04.077553 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:04.077824 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:04.077828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:04.077830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:04.078013 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:04.078427 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:04.078434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:04.078689 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 02:51:04.078858 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:51:04.079237 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260928 02:51:04.081650 23395 heartbeater.cc:347] Connected to a master server at 127.19.139.190:38863
I20260928 02:51:04.081897 23395 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:04.082402 23395 heartbeater.cc:511] Master 127.19.139.190:38863 requested a full tablet report, sending...
I20260928 02:51:04.083530 23070 ts_manager.cc:194] Registered new tserver with Master: 8e7d53fd01fa4257b781dbeacf12f1bf (127.19.139.130:43695)
I20260928 02:51:04.084105 23070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:41515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:04.094681 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38863
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:04.206670 23400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:04.206971 23400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:04.207015 23400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:04.211185 23400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:04.211263 23400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:04.211330 23400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:04.216995 23400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:04.218106 23400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:04.219120 23400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:04.229002 23408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:04.229003 23405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:04.228993 23406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:04.229156 23400 server_base.cc:1034] running on GCE node
I20260928 02:51:04.229774 23400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:04.230465 23400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:04.231618 23400 hybrid_clock.cc:648] HybridClock initialized: now 1790563864231600 us; error 31 us; skew 500 ppm
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563864, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:04.234365 23400 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:04.235249 23400 webserver.cc:460] Webserver started at http://127.19.139.131:36519/ using document root <none> and password file <none>
I20260928 02:51:04.235788 23400 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:04.235867 23400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:04.236085 23400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:04.238250 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "a369808f179947a5951004ca6588f985"
format_stamp: "Formatted at 2026-09-28 02:51:04 on dist-test-slave-5vc0"
server_key: "34a14169ddc767e125095a7f783d8b59"
server_key_iv: "0cee9f119553137dce9e39fbe09812c3"
server_key_version: "encryptionkey@0"
I20260928 02:51:04.238775 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "a369808f179947a5951004ca6588f985"
format_stamp: "Formatted at 2026-09-28 02:51:04 on dist-test-slave-5vc0"
server_key: "34a14169ddc767e125095a7f783d8b59"
server_key_iv: "0cee9f119553137dce9e39fbe09812c3"
server_key_version: "encryptionkey@0"
I20260928 02:51:04.242489 23400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:51:04.244899 23415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:04.246173 23400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:51:04.246279 23400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "a369808f179947a5951004ca6588f985"
format_stamp: "Formatted at 2026-09-28 02:51:04 on dist-test-slave-5vc0"
server_key: "34a14169ddc767e125095a7f783d8b59"
server_key_iv: "0cee9f119553137dce9e39fbe09812c3"
server_key_version: "encryptionkey@0"
I20260928 02:51:04.246393 23400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:04.271626 23400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:04.274904 23400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:04.275084 23400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:04.275671 23400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:04.276563 23400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:04.276609 23400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:04.276651 23400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:04.276679 23400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:04.285189 23400 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:39145
I20260928 02:51:04.285247 23528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:39145 every 8 connection(s)
I20260928 02:51:04.286103 23400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:04.291445 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23400
I20260928 02:51:04.291554 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:04.291822 20014 external_mini_cluster.cc:1428] Setting key 1e8b6b43f7ed4dcb0f2370555217a173
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563864, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:04.299660 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:04.288057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:37593 (local address 127.19.139.190:38863)
0928 02:51:04.288246 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:04.288250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:04.289105 (+ 855us) server_negotiation.cc:407] Connection header received
0928 02:51:04.290496 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:04.290500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:04.290560 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:04.290633 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:04.291930 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.292506 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.293664 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.293874 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.293966 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:04.296358 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:04.296361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:04.296363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:04.296389 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:04.297629 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:04.298127 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:04.298130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:04.298131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:04.298213 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:04.298554 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:04.298557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:04.298559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:04.298749 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:04.298834 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:04.299365 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 02:51:04.299497 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:51:04.300609 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:04.288477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:38863 (local address 127.19.139.131:37593)
0928 02:51:04.288944 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:04.288991 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:04.290262 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:04.290830 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:04.290838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:04.291218 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:04.291772 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:04.291788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.292633 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.292636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:04.293542 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:04.293552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.294011 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.294015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:04.294183 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:04.294716 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:04.294733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:04.296188 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:04.297742 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:04.297748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:04.297765 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:04.298030 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:04.298293 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:04.298296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:04.298298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:04.298463 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:04.298848 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:04.298855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:04.299133 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 02:51:04.299307 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:51:04.299666 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260928 02:51:04.301723 23529 heartbeater.cc:347] Connected to a master server at 127.19.139.190:38863
I20260928 02:51:04.301941 23529 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:04.302399 23529 heartbeater.cc:511] Master 127.19.139.190:38863 requested a full tablet report, sending...
I20260928 02:51:04.303400 23070 ts_manager.cc:194] Registered new tserver with Master: a369808f179947a5951004ca6588f985 (127.19.139.131:39145)
I20260928 02:51:04.303946 23070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:37593
I20260928 02:51:04.305780 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563864, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563864, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:04.331331 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:04.322319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.19.139.190:38863)
0928 02:51:04.322488 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:04.322493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:04.322583 (+ 90us) server_negotiation.cc:407] Connection header received
0928 02:51:04.322700 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:04.322703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:04.322748 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:04.322816 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:04.323518 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.324048 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.324973 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:04.325182 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:04.325259 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:04.328092 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:04.328096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:04.328098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:04.328132 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:04.329517 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:04.329988 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:04.329991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:04.329992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:04.330077 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:04.330413 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:04.330415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:04.330416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:04.330647 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:04.330755 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:04.331032 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:51:04.331164 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:51:04.331772 23070 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52784
I20260928 02:51:04.332685 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23132
I20260928 02:51:04.337361 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23266
I20260928 02:51:04.342124 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23400
I20260928 02:51:04.346494 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23038
2026-09-28T02:51:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2065 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23541](info): set up 2 sockets
Sep 28 02:51:04 dist-test-slave-5vc0 krb5kdc[23541](info): commencing operation
krb5kdc: starting...
W20260928 02:51:05.312330 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:05 dist-test-slave-5vc0 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:05.431317 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40217
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:40217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:05.541632 23554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:05.541923 23554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:05.541981 23554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:05.546041 23554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:05.546108 23554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:05.546135 23554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:05.546156 23554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:05.546177 23554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:05.551640 23554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:40217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:05.552793 23554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:05.553853 23554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:05.562333 23559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:05.562345 23560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:05.562345 23562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:05.562556 23554 server_base.cc:1034] running on GCE node
I20260928 02:51:05.563158 23554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:05.564335 23554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:05.565519 23554 hybrid_clock.cc:648] HybridClock initialized: now 1790563865565501 us; error 30 us; skew 500 ppm
Sep 28 02:51:05 dist-test-slave-5vc0 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563865, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:05.568241 23554 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:05.569121 23554 webserver.cc:460] Webserver started at http://127.19.139.190:46641/ using document root <none> and password file <none>
I20260928 02:51:05.569669 23554 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:05.569734 23554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:05.569921 23554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:05.571947 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "c1e5c6c1ecad48c1bef409e4595714bd"
format_stamp: "Formatted at 2026-09-28 02:51:05 on dist-test-slave-5vc0"
server_key: "92b1af3ac5c4acf25d23b1e112fd34c6"
server_key_iv: "717a1ceeac0b3993da3146c7aeb78cc0"
server_key_version: "encryptionkey@0"
I20260928 02:51:05.572532 23554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "c1e5c6c1ecad48c1bef409e4595714bd"
format_stamp: "Formatted at 2026-09-28 02:51:05 on dist-test-slave-5vc0"
server_key: "92b1af3ac5c4acf25d23b1e112fd34c6"
server_key_iv: "717a1ceeac0b3993da3146c7aeb78cc0"
server_key_version: "encryptionkey@0"
I20260928 02:51:05.576187 23554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:05.578577 23569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:05.579828 23554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:05.579968 23554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "c1e5c6c1ecad48c1bef409e4595714bd"
format_stamp: "Formatted at 2026-09-28 02:51:05 on dist-test-slave-5vc0"
server_key: "92b1af3ac5c4acf25d23b1e112fd34c6"
server_key_iv: "717a1ceeac0b3993da3146c7aeb78cc0"
server_key_version: "encryptionkey@0"
I20260928 02:51:05.580073 23554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:05.603020 23554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:05.606076 23554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:05.606266 23554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:05.613425 23554 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:40217
I20260928 02:51:05.613451 23621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:40217 every 8 connection(s)
I20260928 02:51:05.614385 23554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:05.617143 23622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:05.617386 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23554
I20260928 02:51:05.617507 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:05.617790 20014 external_mini_cluster.cc:1428] Setting key b89b8510efee86d877099bcb38d71eec
I20260928 02:51:05.623780 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd: Bootstrap starting.
I20260928 02:51:05.626351 23622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd: Neither blocks nor log segments found. Creating new log.
Sep 28 02:51:05 dist-test-slave-5vc0 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:05.627065 23622 log.cc:824] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:05.628870 23622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd: No bootstrap required, opened a new log
I20260928 02:51:05.631383 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:05.619115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.19.139.190:40217)
0928 02:51:05.619489 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:05.619505 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:05.619539 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:51:05.620351 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:05.620372 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:05.620746 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:05.621037 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:05.621866 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:05.622922 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:05.623994 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:05.624498 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:05.624609 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:05.627154 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:05.627159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:05.627174 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:05.627207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:05.628781 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:05.629211 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:05.629215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:05.629219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:05.629330 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:05.629593 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:05.629596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:05.629598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:05.629977 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:05.630140 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:05.630422 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:51:05.630690 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260928 02:51:05.632367 23622 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40217 } }
I20260928 02:51:05.632632 23622 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:05.632692 23622 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e5c6c1ecad48c1bef409e4595714bd, State: Initialized, Role: FOLLOWER
I20260928 02:51:05.633147 23622 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40217 } }
I20260928 02:51:05.633292 23622 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:05.633351 23622 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:05.633445 23622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:05.634284 23622 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40217 } }
I20260928 02:51:05.634588 23622 leader_election.cc:302] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e5c6c1ecad48c1bef409e4595714bd; no voters:
I20260928 02:51:05.634891 23622 leader_election.cc:288] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:05.634979 23627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:05.635191 23627 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [term 1 LEADER]: Becoming Leader. State: Replica: c1e5c6c1ecad48c1bef409e4595714bd, State: Running, Role: LEADER
I20260928 02:51:05.635515 23627 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40217 } }
I20260928 02:51:05.636005 23622 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:05.637233 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40217 } } }
I20260928 02:51:05.637229 23629 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e5c6c1ecad48c1bef409e4595714bd. Latest consensus state: current_term: 1 leader_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e5c6c1ecad48c1bef409e4595714bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40217 } } }
I20260928 02:51:05.637391 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:05.637615 23629 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:05.637799 23637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:05.641264 23637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:05.647163 23637 catalog_manager.cc:1347] Generated new cluster ID: 12f7c787f95143c78ba7d2fcb48a8d3e
I20260928 02:51:05.647233 23637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:05.657603 23637 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:05.658536 23637 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:05.667932 23637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1e5c6c1ecad48c1bef409e4595714bd: Generated new TSK 0
I20260928 02:51:05.668666 23637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:05.702915 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40217
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:05.841416 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:05.841701 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:05.841743 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:05.845888 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:05.845961 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:05.846029 23648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:05.851477 23648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:05.852625 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:05.853669 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:51:05.863811 23648 server_base.cc:1034] running on GCE node
W20260928 02:51:05.863760 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:05.863746 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:05.863969 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:05.864478 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:05.865103 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:05.866271 23648 hybrid_clock.cc:648] HybridClock initialized: now 1790563865866249 us; error 32 us; skew 500 ppm
Sep 28 02:51:05 dist-test-slave-5vc0 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563865, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:05.869047 23648 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:05.869969 23648 webserver.cc:460] Webserver started at http://127.19.139.129:36201/ using document root <none> and password file <none>
I20260928 02:51:05.870510 23648 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:05.870576 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:05.870777 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:05.872889 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "3da7d30f81f94a80af195a50d47e59e6"
format_stamp: "Formatted at 2026-09-28 02:51:05 on dist-test-slave-5vc0"
server_key: "67d1dc3ff3f2dfbb7c8a96bed18609aa"
server_key_iv: "aca13f3e37e1ebe9c42ef76e39a28a9a"
server_key_version: "encryptionkey@0"
I20260928 02:51:05.873453 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "3da7d30f81f94a80af195a50d47e59e6"
format_stamp: "Formatted at 2026-09-28 02:51:05 on dist-test-slave-5vc0"
server_key: "67d1dc3ff3f2dfbb7c8a96bed18609aa"
server_key_iv: "aca13f3e37e1ebe9c42ef76e39a28a9a"
server_key_version: "encryptionkey@0"
I20260928 02:51:05.877184 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:05.879578 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:05.880892 23648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:05.880998 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "3da7d30f81f94a80af195a50d47e59e6"
format_stamp: "Formatted at 2026-09-28 02:51:05 on dist-test-slave-5vc0"
server_key: "67d1dc3ff3f2dfbb7c8a96bed18609aa"
server_key_iv: "aca13f3e37e1ebe9c42ef76e39a28a9a"
server_key_version: "encryptionkey@0"
I20260928 02:51:05.881095 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:05.892627 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:05.895665 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:05.895835 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:05.896483 23648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:05.897274 23648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:05.897318 23648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:05.897369 23648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:05.897395 23648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:05.905624 23648 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:45719
I20260928 02:51:05.905705 23776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:45719 every 8 connection(s)
I20260928 02:51:05.906529 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:05.910071 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23648
I20260928 02:51:05.910164 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:05.910430 20014 external_mini_cluster.cc:1428] Setting key 4dfbf615d9d8f59156a0bc94fbac2380
Sep 28 02:51:05 dist-test-slave-5vc0 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563865, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:05.920723 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:05.908276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:45919 (local address 127.19.139.190:40217)
0928 02:51:05.908419 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:05.908423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:05.909313 (+ 890us) server_negotiation.cc:407] Connection header received
0928 02:51:05.910704 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:05.910708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:05.910762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:05.910835 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:05.912117 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:05.912907 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:05.914078 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:05.914420 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:05.914505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:05.917263 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:05.917268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:05.917272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:05.917305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:05.918789 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:05.919256 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:05.919260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:05.919262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:05.919347 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:05.919661 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:05.919665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:05.919666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:05.919899 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:05.920001 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:05.920488 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 02:51:05.920597 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:51:05.921605 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:05.908720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40217 (local address 127.19.139.129:45919)
0928 02:51:05.909160 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:05.909205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:05.910505 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:05.911050 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:05.911059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:05.911421 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:05.911975 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:05.911990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:05.913035 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:05.913038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:05.913958 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:05.913968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:05.914578 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:05.914582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:05.914781 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:05.915434 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:05.915455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:05.917090 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:05.918895 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:05.918902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:05.918915 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:05.919161 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:05.919448 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:05.919451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:05.919452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:05.919584 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:05.919995 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:05.920000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:05.920266 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 02:51:05.920437 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:51:05.920695 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:51:05.922940 23777 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40217
I20260928 02:51:05.923174 23777 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:05.923676 23777 heartbeater.cc:511] Master 127.19.139.190:40217 requested a full tablet report, sending...
I20260928 02:51:05.925293 23586 ts_manager.cc:194] Registered new tserver with Master: 3da7d30f81f94a80af195a50d47e59e6 (127.19.139.129:45719)
I20260928 02:51:05.926601 23586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:45919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:05.938869 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40217
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:06.048076 23782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:06.048394 23782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:06.048458 23782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:06.052609 23782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:06.052686 23782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:06.052768 23782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:06.058279 23782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:06.059432 23782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:06.060694 23782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:06.070688 23788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:06.070681 23790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:06.070700 23787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:06.070742 23782 server_base.cc:1034] running on GCE node
I20260928 02:51:06.071343 23782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:06.072041 23782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:06.073195 23782 hybrid_clock.cc:648] HybridClock initialized: now 1790563866073185 us; error 70 us; skew 500 ppm
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:06.076027 23782 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:06.076984 23782 webserver.cc:460] Webserver started at http://127.19.139.130:33785/ using document root <none> and password file <none>
I20260928 02:51:06.077527 23782 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:06.077593 23782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:06.077788 23782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:06.079881 23782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "1338fc2ed266472ea927f4090c70c70e"
format_stamp: "Formatted at 2026-09-28 02:51:06 on dist-test-slave-5vc0"
server_key: "d3c1942bff3385023928ce2c016c56bd"
server_key_iv: "7943e3e81ca7e4fcffeccc6da4e2a7ed"
server_key_version: "encryptionkey@0"
I20260928 02:51:06.080472 23782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "1338fc2ed266472ea927f4090c70c70e"
format_stamp: "Formatted at 2026-09-28 02:51:06 on dist-test-slave-5vc0"
server_key: "d3c1942bff3385023928ce2c016c56bd"
server_key_iv: "7943e3e81ca7e4fcffeccc6da4e2a7ed"
server_key_version: "encryptionkey@0"
I20260928 02:51:06.084566 23782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:06.087109 23797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:06.088514 23782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:06.088626 23782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "1338fc2ed266472ea927f4090c70c70e"
format_stamp: "Formatted at 2026-09-28 02:51:06 on dist-test-slave-5vc0"
server_key: "d3c1942bff3385023928ce2c016c56bd"
server_key_iv: "7943e3e81ca7e4fcffeccc6da4e2a7ed"
server_key_version: "encryptionkey@0"
I20260928 02:51:06.088727 23782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:06.105197 23782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:06.108902 23782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:06.109093 23782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:06.109691 23782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:06.110528 23782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:06.110576 23782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:06.110626 23782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:06.110656 23782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:06.118950 23782 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:32987
I20260928 02:51:06.118984 23910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:32987 every 8 connection(s)
I20260928 02:51:06.119897 23782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:06.125185 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23782
I20260928 02:51:06.125290 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:06.125535 20014 external_mini_cluster.cc:1428] Setting key f9ebbe01d519af281302e4062b467c97
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:06.134320 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:06.121820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:36337 (local address 127.19.139.190:40217)
0928 02:51:06.121959 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:06.121962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:06.122890 (+ 928us) server_negotiation.cc:407] Connection header received
0928 02:51:06.124230 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:06.124233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:06.124282 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:06.124364 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:06.125594 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.126175 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.127403 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.127622 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.127708 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:06.130381 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:06.130384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:06.130387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:06.130419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:06.131960 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.132621 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.132624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.132625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.132701 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.133085 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.133087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.133088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.133277 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:06.133354 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:06.134050 (+ 696us) server_negotiation.cc:299] Negotiation successful
0928 02:51:06.134176 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:51:06.135476 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:06.122183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40217 (local address 127.19.139.130:36337)
0928 02:51:06.122718 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:06.122762 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:06.124004 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:06.124522 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:06.124529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:06.124907 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:06.125439 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:06.125455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.126283 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.126287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:06.127284 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:06.127295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.127760 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.127764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:06.127933 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:06.128519 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:06.128536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:06.130174 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:06.132095 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:06.132102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:06.132122 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:06.132527 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:06.132789 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:06.132793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:06.132795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:06.132990 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:06.133377 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:06.133385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:06.133740 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 02:51:06.133979 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 02:51:06.134354 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":131,"threads_started":1}
I20260928 02:51:06.136866 23911 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40217
I20260928 02:51:06.137099 23911 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:06.137722 23911 heartbeater.cc:511] Master 127.19.139.190:40217 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:51:06.138859 23586 ts_manager.cc:194] Registered new tserver with Master: 1338fc2ed266472ea927f4090c70c70e (127.19.139.130:32987)
I20260928 02:51:06.139495 23586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:36337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:06.155019 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40217
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:06.264438 23916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:06.264722 23916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:06.264793 23916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:06.268947 23916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:06.269019 23916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:06.269101 23916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:06.275473 23916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:06.276691 23916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:06.277752 23916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:51:06.287973 23916 server_base.cc:1034] running on GCE node
W20260928 02:51:06.287923 23921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:06.287921 23922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:06.287946 23924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:06.288632 23916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:06.289295 23916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:06.290473 23916 hybrid_clock.cc:648] HybridClock initialized: now 1790563866290447 us; error 40 us; skew 500 ppm
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:06.293273 23916 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:06.294229 23916 webserver.cc:460] Webserver started at http://127.19.139.131:44007/ using document root <none> and password file <none>
I20260928 02:51:06.294780 23916 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:06.294842 23916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:06.295042 23916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:06.297207 23916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "aae37523f62b4c75b14db29054026f2b"
format_stamp: "Formatted at 2026-09-28 02:51:06 on dist-test-slave-5vc0"
server_key: "a008e8c1a84636821bd8d5f6fcdc0bb2"
server_key_iv: "2e9fb9cacebc646e176e1227362aa162"
server_key_version: "encryptionkey@0"
I20260928 02:51:06.297767 23916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "aae37523f62b4c75b14db29054026f2b"
format_stamp: "Formatted at 2026-09-28 02:51:06 on dist-test-slave-5vc0"
server_key: "a008e8c1a84636821bd8d5f6fcdc0bb2"
server_key_iv: "2e9fb9cacebc646e176e1227362aa162"
server_key_version: "encryptionkey@0"
I20260928 02:51:06.301573 23916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:06.304013 23931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:06.305300 23916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:06.305449 23916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "aae37523f62b4c75b14db29054026f2b"
format_stamp: "Formatted at 2026-09-28 02:51:06 on dist-test-slave-5vc0"
server_key: "a008e8c1a84636821bd8d5f6fcdc0bb2"
server_key_iv: "2e9fb9cacebc646e176e1227362aa162"
server_key_version: "encryptionkey@0"
I20260928 02:51:06.305553 23916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:06.317397 23916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:06.320739 23916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:06.320921 23916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:06.321533 23916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:06.322395 23916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:06.322481 23916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:06.322528 23916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:06.322557 23916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:06.330935 23916 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:43675
I20260928 02:51:06.330973 24044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:43675 every 8 connection(s)
I20260928 02:51:06.331828 23916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:06.341583 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 23916
I20260928 02:51:06.341710 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:06.342000 20014 external_mini_cluster.cc:1428] Setting key 8a22c2eb826c1ca831f2ffdcd6f62198
I20260928 02:51:06.345813 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:06.333635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:47579 (local address 127.19.139.190:40217)
0928 02:51:06.333756 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:06.333759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:06.334611 (+ 852us) server_negotiation.cc:407] Connection header received
0928 02:51:06.336079 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:06.336083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:06.336145 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:06.336232 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:06.337594 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.338197 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.339361 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.339594 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.339704 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:06.342510 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:06.342513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:06.342515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:06.342541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:06.343943 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.344496 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.344499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.344500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.344583 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.344848 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.344850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.344851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.345029 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:06.345099 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:06.345573 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 02:51:06.345684 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:51:06.346684 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:06.334012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40217 (local address 127.19.139.131:47579)
0928 02:51:06.334457 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:06.334502 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:06.335840 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:06.336437 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:06.336445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:06.336891 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:06.337446 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:06.337461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.338325 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.338327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:06.339232 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:06.339242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.339831 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.339840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:06.340023 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:06.340804 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:06.340821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:06.342318 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:06.344063 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:06.344069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:06.344086 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:06.344401 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:06.344652 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:06.344654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:06.344656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:06.344787 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:06.345102 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:06.345107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:06.345353 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:51:06.345545 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:51:06.345839 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260928 02:51:06.347785 24045 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40217
I20260928 02:51:06.348033 24045 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:06.348551 24045 heartbeater.cc:511] Master 127.19.139.190:40217 requested a full tablet report, sending...
I20260928 02:51:06.349588 23586 ts_manager.cc:194] Registered new tserver with Master: aae37523f62b4c75b14db29054026f2b (127.19.139.131:43675)
I20260928 02:51:06.350133 23586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:47579
I20260928 02:51:06.355718 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:51:06.380541 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:06.371523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.19.139.129:45719)
0928 02:51:06.371893 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:06.371903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:06.371930 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:51:06.371980 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:06.371986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:06.372220 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:06.372348 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:06.373024 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.373819 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.374621 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.375004 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.375081 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:06.377408 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:06.377424 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:06.377439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:06.377464 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:06.378860 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.379288 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.379292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.379294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.379372 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.379573 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.379575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.379577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.379811 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:06.379963 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:06.380130 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:51:06.380322 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":147,"threads_started":1}
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[23541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:06.390831 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:06.383354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.19.139.190:40217)
0928 02:51:06.383521 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:06.383526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:06.383591 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:51:06.383691 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:06.383693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:06.383732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:06.383795 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:06.384359 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.384968 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.385770 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:06.386028 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:06.386105 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:06.388231 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:06.388234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:06.388236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:06.388260 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:06.389470 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.389799 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.389801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.389802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.389868 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:06.390043 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:06.390045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:06.390046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:06.390253 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:06.390354 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:06.390574 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:51:06.390672 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:51:06.391234 23586 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56684
I20260928 02:51:06.391870 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23648
I20260928 02:51:06.396271 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23782
I20260928 02:51:06.400794 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23916
I20260928 02:51:06.405342 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 23554
2026-09-28T02:51:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2058 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[24057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[24057](info): set up 2 sockets
Sep 28 02:51:06 dist-test-slave-5vc0 krb5kdc[24057](info): commencing operation
krb5kdc: starting...
W20260928 02:51:07.360100 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:07 dist-test-slave-5vc0 krb5kdc[24057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:07.480835 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:33511
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:07.590919 24070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:07.591230 24070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:07.591291 24070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:07.595433 24070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:07.595510 24070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:07.595537 24070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:07.595558 24070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:07.595580 24070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:07.601071 24070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:33511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:07.602213 24070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:07.603295 24070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:07.611614 24075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:07.611622 24076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:07.611631 24078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:07.611815 24070 server_base.cc:1034] running on GCE node
I20260928 02:51:07.612540 24070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:07.613750 24070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:07.614925 24070 hybrid_clock.cc:648] HybridClock initialized: now 1790563867614904 us; error 38 us; skew 500 ppm
Sep 28 02:51:07 dist-test-slave-5vc0 krb5kdc[24057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:07.617808 24070 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:07.618577 24070 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:07.618649 24070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:07.618851 24070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:07.621060 24070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "b4fe3808505e4f23baf571dde468f766"
format_stamp: "Formatted at 2026-09-28 02:51:07 on dist-test-slave-5vc0"
server_key: "d572888826be5740b0bb2bae16b7ae02"
server_key_iv: "3446303bd73cf989993aec08bd1975ff"
server_key_version: "encryptionkey@0"
I20260928 02:51:07.621647 24070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "b4fe3808505e4f23baf571dde468f766"
format_stamp: "Formatted at 2026-09-28 02:51:07 on dist-test-slave-5vc0"
server_key: "d572888826be5740b0bb2bae16b7ae02"
server_key_iv: "3446303bd73cf989993aec08bd1975ff"
server_key_version: "encryptionkey@0"
I20260928 02:51:07.625468 24070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:07.627867 24084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:07.629115 24070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:07.629235 24070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "b4fe3808505e4f23baf571dde468f766"
format_stamp: "Formatted at 2026-09-28 02:51:07 on dist-test-slave-5vc0"
server_key: "d572888826be5740b0bb2bae16b7ae02"
server_key_iv: "3446303bd73cf989993aec08bd1975ff"
server_key_version: "encryptionkey@0"
I20260928 02:51:07.629329 24070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:07.651582 24070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:07.654799 24070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:07.654991 24070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:07.661823 24070 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:33511
I20260928 02:51:07.662068 24136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:33511 every 8 connection(s)
I20260928 02:51:07.662472 24070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:07.665207 24137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:07.667436 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24070
I20260928 02:51:07.667567 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:07.667853 20014 external_mini_cluster.cc:1428] Setting key ff58a2a20c947d6a9a9101843c9d8428
I20260928 02:51:07.671425 24137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766: Bootstrap starting.
I20260928 02:51:07.673947 24137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:07.674698 24137 log.cc:824] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:51:07 dist-test-slave-5vc0 krb5kdc[24057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:07.676757 24137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766: No bootstrap required, opened a new log
I20260928 02:51:07.681227 24137 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fe3808505e4f23baf571dde468f766" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33511 } }
I20260928 02:51:07.681516 24137 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:07.681571 24137 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fe3808505e4f23baf571dde468f766, State: Initialized, Role: FOLLOWER
I20260928 02:51:07.682096 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:07.669237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.19.139.190:33511)
0928 02:51:07.669617 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:07.669632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:07.669666 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:51:07.670469 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:07.670487 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:07.670820 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:07.671111 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:07.671973 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:07.672971 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:07.674082 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:07.674581 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:07.674696 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:07.677074 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:07.677082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:07.677100 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:07.677130 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:07.678980 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:07.679413 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:07.679419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:07.679424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:07.679565 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:07.679778 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:07.679781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:07.679783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:07.680352 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:07.680559 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:07.680785 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:51:07.681105 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260928 02:51:07.682037 24137 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fe3808505e4f23baf571dde468f766" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33511 } }
I20260928 02:51:07.682206 24137 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:07.682271 24137 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:07.682367 24137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:07.683506 24137 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fe3808505e4f23baf571dde468f766" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33511 } }
I20260928 02:51:07.683837 24137 leader_election.cc:302] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4fe3808505e4f23baf571dde468f766; no voters:
I20260928 02:51:07.684139 24137 leader_election.cc:288] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:07.684264 24142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:07.684501 24142 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [term 1 LEADER]: Becoming Leader. State: Replica: b4fe3808505e4f23baf571dde468f766, State: Running, Role: LEADER
I20260928 02:51:07.684826 24142 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fe3808505e4f23baf571dde468f766" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33511 } }
I20260928 02:51:07.685391 24137 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:07.686396 24144 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4fe3808505e4f23baf571dde468f766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fe3808505e4f23baf571dde468f766" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33511 } } }
I20260928 02:51:07.686430 24143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4fe3808505e4f23baf571dde468f766. Latest consensus state: current_term: 1 leader_uuid: "b4fe3808505e4f23baf571dde468f766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fe3808505e4f23baf571dde468f766" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33511 } } }
I20260928 02:51:07.686563 24143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:07.686560 24144 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:07.686961 24151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:07.690192 24151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:07.696189 24151 catalog_manager.cc:1347] Generated new cluster ID: 765eb60145414cef891049b8faef3f21
I20260928 02:51:07.696270 24151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:07.712584 24151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:07.713641 24151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:07.722941 24151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766: Generated new TSK 0
I20260928 02:51:07.723711 24151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:07.758962 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33511
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:07.902343 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:07.902629 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:07.902673 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:07.906689 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:07.906751 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:07.906812 24163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:07.912214 24163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:07.913381 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:07.914466 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:07.924077 24171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:07.924132 24169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:07.924100 24168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:07.924454 24163 server_base.cc:1034] running on GCE node
I20260928 02:51:07.924894 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:07.925589 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:07.926813 24163 hybrid_clock.cc:648] HybridClock initialized: now 1790563867926790 us; error 34 us; skew 500 ppm
Sep 28 02:51:07 dist-test-slave-5vc0 krb5kdc[24057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:07.929651 24163 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:07.930433 24163 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:07.930505 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:07.930708 24163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:07.933033 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "573f11efdc3e4aeda6a882491452af19"
format_stamp: "Formatted at 2026-09-28 02:51:07 on dist-test-slave-5vc0"
server_key: "5e370d7b230e070d33108c022831b00b"
server_key_iv: "cd2e420cdc0c73825a5e6b04b36a89ba"
server_key_version: "encryptionkey@0"
I20260928 02:51:07.933699 24163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "573f11efdc3e4aeda6a882491452af19"
format_stamp: "Formatted at 2026-09-28 02:51:07 on dist-test-slave-5vc0"
server_key: "5e370d7b230e070d33108c022831b00b"
server_key_iv: "cd2e420cdc0c73825a5e6b04b36a89ba"
server_key_version: "encryptionkey@0"
I20260928 02:51:07.937677 24163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:07.940362 24177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:07.941715 24163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:07.941838 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "573f11efdc3e4aeda6a882491452af19"
format_stamp: "Formatted at 2026-09-28 02:51:07 on dist-test-slave-5vc0"
server_key: "5e370d7b230e070d33108c022831b00b"
server_key_iv: "cd2e420cdc0c73825a5e6b04b36a89ba"
server_key_version: "encryptionkey@0"
I20260928 02:51:07.941951 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:07.964433 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:07.967576 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:07.967767 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:07.968387 24163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:07.969235 24163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:07.969282 24163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:07.969336 24163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:07.969372 24163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:07.977967 24163 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:41729
I20260928 02:51:07.977983 24290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:41729 every 8 connection(s)
I20260928 02:51:07.978569 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:07.986946 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24163
I20260928 02:51:07.987054 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:07.987334 20014 external_mini_cluster.cc:1428] Setting key 741d275109242d27193aa628021b9a21
Sep 28 02:51:07 dist-test-slave-5vc0 krb5kdc[24057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:07.992703 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:07.980274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:44095 (local address 127.19.139.190:33511)
0928 02:51:07.980440 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:07.980443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:07.981301 (+ 858us) server_negotiation.cc:407] Connection header received
0928 02:51:07.982545 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:07.982548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:07.982594 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:07.982665 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:07.983848 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:07.984466 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:07.985654 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:07.985901 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:07.985975 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:07.989119 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:07.989122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:07.989124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:07.989147 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:07.990507 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:07.991118 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:07.991121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:07.991123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:07.991212 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:07.991539 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:07.991542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:07.991544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:07.991762 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:07.991859 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:07.992392 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 02:51:07.992528 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:51:07.993554 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:07.980717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33511 (local address 127.19.139.129:44095)
0928 02:51:07.981153 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:07.981195 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:07.982351 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:07.982817 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:07.982824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:07.983181 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:07.983724 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:07.983737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:07.984584 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:07.984587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:07.985545 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:07.985555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:07.986036 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:07.986039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:07.986181 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:07.987212 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:07.987229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:07.988961 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:07.990619 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:07.990626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:07.990644 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:07.991006 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:07.991299 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:07.991302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:07.991303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:07.991456 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:07.991850 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:07.991855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:07.992120 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:51:07.992330 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 02:51:07.992650 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260928 02:51:07.994838 24291 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33511
I20260928 02:51:07.995111 24291 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:07.995675 24291 heartbeater.cc:511] Master 127.19.139.190:33511 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:07.997334 24101 ts_manager.cc:194] Registered new tserver with Master: 573f11efdc3e4aeda6a882491452af19 (127.19.139.129:41729)
I20260928 02:51:07.998442 24101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:44095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:08.014878 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33511
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:08.126907 24296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:08.127210 24296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:08.127272 24296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:08.131508 24296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:08.131585 24296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:08.131675 24296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:08.137727 24296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:08.138865 24296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:08.139940 24296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:08.149737 24302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:08.149721 24304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:08.149807 24296 server_base.cc:1034] running on GCE node
W20260928 02:51:08.149714 24301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:08.150434 24296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:08.151077 24296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:08.152230 24296 hybrid_clock.cc:648] HybridClock initialized: now 1790563868152207 us; error 33 us; skew 500 ppm
Sep 28 02:51:08 dist-test-slave-5vc0 krb5kdc[24057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:08.155126 24296 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:08.155887 24296 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:08.155964 24296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:08.156140 24296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:08.158480 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "1d7477106d6b43018ce77428c5435adb"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "790d17a3c724fc16b98d28261ade0686"
server_key_iv: "6027cdf21d0fc134ba0dd3250d8b0023"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.159044 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "1d7477106d6b43018ce77428c5435adb"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "790d17a3c724fc16b98d28261ade0686"
server_key_iv: "6027cdf21d0fc134ba0dd3250d8b0023"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.162788 24296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:51:08.165529 24310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.166831 24296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:08.166935 24296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "1d7477106d6b43018ce77428c5435adb"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "790d17a3c724fc16b98d28261ade0686"
server_key_iv: "6027cdf21d0fc134ba0dd3250d8b0023"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.167047 24296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:08.189289 24296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:08.192572 24296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:08.192757 24296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:08.193329 24296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:08.194151 24296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:08.194197 24296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.194242 24296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:08.194271 24296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.202680 24296 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:33571
I20260928 02:51:08.202698 24423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:33571 every 8 connection(s)
I20260928 02:51:08.203287 24296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:08.211591 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24296
I20260928 02:51:08.211724 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:08.212040 20014 external_mini_cluster.cc:1428] Setting key 53273d89ed0ed63c93a7020c30f42cac
Sep 28 02:51:08 dist-test-slave-5vc0 krb5kdc[24057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:08.217382 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.205260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:43783 (local address 127.19.139.190:33511)
0928 02:51:08.205404 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.205409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.206379 (+ 970us) server_negotiation.cc:407] Connection header received
0928 02:51:08.207648 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.207651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.207706 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.207777 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:08.209094 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.209680 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.210897 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.211126 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.211217 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.213923 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:08.213926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:08.213928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:08.213955 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:08.215315 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:08.215827 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:08.215830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:08.215831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:08.215906 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:08.216251 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:08.216254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:08.216256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:08.216459 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:08.216548 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.217082 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.217215 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:51:08.218256 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.205719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33511 (local address 127.19.139.130:43783)
0928 02:51:08.206218 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.206265 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.207442 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.207924 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.207931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.208359 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:08.208943 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.208959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.209814 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.209817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.210771 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.210781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.211280 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.211285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.211543 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.212188 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:08.212211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:08.213744 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:08.215430 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:08.215437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:08.215456 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:08.215729 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:08.215997 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:08.216000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:08.216002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:08.216145 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:08.216561 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:08.216569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:08.216839 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.217009 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.217376 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":111,"threads_started":1}
I20260928 02:51:08.219636 24424 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33511
I20260928 02:51:08.219889 24424 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:08.220435 24424 heartbeater.cc:511] Master 127.19.139.190:33511 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:51:08.221547 24101 ts_manager.cc:194] Registered new tserver with Master: 1d7477106d6b43018ce77428c5435adb (127.19.139.130:33571)
I20260928 02:51:08.222121 24101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:43783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:08.238885 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33511
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:43797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:08.350030 24429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:08.350301 24429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:08.350349 24429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:08.354547 24429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:08.354625 24429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:08.354694 24429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:08.360283 24429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:43797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:08.361537 24429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:08.362564 24429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:08.372251 24437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:08.372256 24435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:08.372251 24434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:08.372442 24429 server_base.cc:1034] running on GCE node
I20260928 02:51:08.373032 24429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:08.373744 24429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:08.374905 24429 hybrid_clock.cc:648] HybridClock initialized: now 1790563868374888 us; error 46 us; skew 500 ppm
Sep 28 02:51:08 dist-test-slave-5vc0 krb5kdc[24057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:08.377703 24429 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:08.378463 24429 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:08.378532 24429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:08.378731 24429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:08.380966 24429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "2705da223a084ed1acc62eae37677768"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "f7c9f94b48b769f0f445daf7a109bd6f"
server_key_iv: "f8bc6df87bc7fd4067d95b2291101662"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.381522 24429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "2705da223a084ed1acc62eae37677768"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "f7c9f94b48b769f0f445daf7a109bd6f"
server_key_iv: "f8bc6df87bc7fd4067d95b2291101662"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.385202 24429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:08.387763 24443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.389055 24429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:08.389165 24429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "2705da223a084ed1acc62eae37677768"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "f7c9f94b48b769f0f445daf7a109bd6f"
server_key_iv: "f8bc6df87bc7fd4067d95b2291101662"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.389298 24429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:08.400700 24429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:08.403998 24429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:08.404296 24429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:08.404963 24429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:08.405871 24429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:08.405920 24429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.405974 24429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:08.406002 24429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.415048 24429 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:36247
I20260928 02:51:08.415069 24556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:36247 every 8 connection(s)
I20260928 02:51:08.415784 24429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:08.425253 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24429
Sep 28 02:51:08 dist-test-slave-5vc0 krb5kdc[24057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:08.425405 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:08.425669 20014 external_mini_cluster.cc:1428] Setting key dde3d361629d43dade6ff0dd8b239745
I20260928 02:51:08.429419 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.417670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:36767 (local address 127.19.139.190:33511)
0928 02:51:08.417824 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.417828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.419066 (+ 1238us) server_negotiation.cc:407] Connection header received
0928 02:51:08.419968 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.419971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.420021 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.420085 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:08.421391 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.421944 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.423133 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.423342 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.423420 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.426058 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:08.426062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:08.426065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:08.426097 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:08.427489 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:08.428016 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:08.428019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:08.428021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:08.428103 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:08.428474 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:08.428477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:08.428478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:08.428660 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:08.428732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.429200 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.429296 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:51:08.430274 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.418102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33511 (local address 127.19.139.131:36767)
0928 02:51:08.418912 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.418957 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.419738 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.420302 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.420311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.420701 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:08.421246 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.421261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.422075 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.422078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.423033 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.423043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.423486 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.423488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.423638 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.424279 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:08.424296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:08.425860 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:08.427609 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:08.427615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:08.427632 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:08.427939 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:08.428199 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:08.428202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:08.428204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:08.428376 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:08.428735 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:08.428740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:08.429008 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.429161 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.429437 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260928 02:51:08.431324 24557 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33511
I20260928 02:51:08.431586 24557 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:08.432039 24557 heartbeater.cc:511] Master 127.19.139.190:33511 requested a full tablet report, sending...
I20260928 02:51:08.433112 24101 ts_manager.cc:194] Registered new tserver with Master: 2705da223a084ed1acc62eae37677768 (127.19.139.131:36247)
I20260928 02:51:08.433682 24101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:36767
I20260928 02:51:08.439847 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:51:08.449134 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.441902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.19.139.190:33511)
0928 02:51:08.442038 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.442043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.442161 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:51:08.442306 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.442309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.442358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.442428 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:08.443107 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.443662 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.444655 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.444869 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.444951 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.446287 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:08.446290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:08.446292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:08.446318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:08.447513 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:08.447874 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:08.447877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:08.447879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:08.447952 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:08.448222 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:08.448226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:08.448228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:08.448495 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:08.448608 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.448894 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.449007 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:51:08.452069 24101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:08.455338 24101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:08.470413 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.464464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33571 (local address 127.0.0.1:39492)
0928 02:51:08.464839 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.464894 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.465313 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.466282 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.466288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.466310 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:08.466478 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.466485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.467887 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.467891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.468985 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.468994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.469832 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.469836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.469962 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.470003 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.470112 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.470242 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":17,"thread_start_us":93,"threads_started":1}
I20260928 02:51:08.470419 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.464574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.19.139.130:33571)
0928 02:51:08.465777 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.465789 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.465811 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:51:08.465894 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.465899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.466143 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.466289 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:08.466597 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.467739 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.469104 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.469711 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.469830 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.469985 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.470151 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.470249 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":103,"threads_started":1}
I20260928 02:51:08.470759 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.465301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:36247 (local address 127.0.0.1:53412)
0928 02:51:08.465981 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.465993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.466071 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.466742 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.466744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.466762 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:08.466844 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.466850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.468279 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.468282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.469053 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.469064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.470361 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.470365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.470457 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.470480 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.470544 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.470616 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":25,"thread_start_us":545,"threads_started":1}
I20260928 02:51:08.471365 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.464537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:41729 (local address 127.0.0.1:38978)
0928 02:51:08.465656 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.465670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.466059 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.466557 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.466560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.466577 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:08.466666 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.466671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.467681 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.467685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.469816 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.469826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.470994 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.470998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.471097 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.471119 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.471178 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.471241 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260928 02:51:08.471483 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.465573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.19.139.129:41729)
0928 02:51:08.465951 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.465958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.466150 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:51:08.466228 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.466233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.466422 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.466571 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:08.466753 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.467544 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.470039 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.470904 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.471026 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.471145 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.471198 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.471276 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":72,"threads_started":1}
I20260928 02:51:08.471607 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.465476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.19.139.131:36247)
0928 02:51:08.466142 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.466153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.466173 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:51:08.466233 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.466237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.466422 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.466572 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:08.466969 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.468108 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.469187 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.470074 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.470179 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.470300 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.471003 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.471173 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":172,"threads_started":1}
I20260928 02:51:08.473691 24358 tablet_service.cc:1467] Processing CreateTablet for tablet 73de866e581f4499bd53b547760fb894 (DEFAULT_TABLE table=test-table [id=023db72b6ae040b58af1eff313ad5466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:08.474738 24358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73de866e581f4499bd53b547760fb894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:08.474380 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 73de866e581f4499bd53b547760fb894 (DEFAULT_TABLE table=test-table [id=023db72b6ae040b58af1eff313ad5466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:08.474381 24225 tablet_service.cc:1467] Processing CreateTablet for tablet 73de866e581f4499bd53b547760fb894 (DEFAULT_TABLE table=test-table [id=023db72b6ae040b58af1eff313ad5466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:08.475467 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73de866e581f4499bd53b547760fb894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:08.475467 24225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73de866e581f4499bd53b547760fb894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:08.481475 24574 tablet_bootstrap.cc:492] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: Bootstrap starting.
I20260928 02:51:08.482657 24575 tablet_bootstrap.cc:492] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: Bootstrap starting.
I20260928 02:51:08.483753 24574 tablet_bootstrap.cc:654] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:08.483795 24576 tablet_bootstrap.cc:492] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Bootstrap starting.
I20260928 02:51:08.484988 24575 tablet_bootstrap.cc:654] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:08.485076 24574 log.cc:824] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:08.485920 24576 tablet_bootstrap.cc:654] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:08.486192 24575 log.cc:824] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:08.487061 24576 log.cc:824] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:08.487308 24574 tablet_bootstrap.cc:492] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: No bootstrap required, opened a new log
I20260928 02:51:08.487555 24574 ts_tablet_manager.cc:1397] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 02:51:08.488210 24575 tablet_bootstrap.cc:492] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: No bootstrap required, opened a new log
I20260928 02:51:08.488469 24575 ts_tablet_manager.cc:1397] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:51:08.489245 24576 tablet_bootstrap.cc:492] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: No bootstrap required, opened a new log
I20260928 02:51:08.489410 24576 ts_tablet_manager.cc:1397] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:51:08.491510 24574 raft_consensus.cc:348] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.491812 24574 raft_consensus.cc:374] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:08.491907 24574 raft_consensus.cc:729] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7477106d6b43018ce77428c5435adb, State: Initialized, Role: FOLLOWER
I20260928 02:51:08.492110 24576 raft_consensus.cc:348] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.492121 24575 raft_consensus.cc:348] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.492444 24576 raft_consensus.cc:374] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:08.492414 24574 consensus_queue.cc:260] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.492511 24575 raft_consensus.cc:374] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:08.492520 24576 raft_consensus.cc:729] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2705da223a084ed1acc62eae37677768, State: Initialized, Role: FOLLOWER
I20260928 02:51:08.492558 24575 raft_consensus.cc:729] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573f11efdc3e4aeda6a882491452af19, State: Initialized, Role: FOLLOWER
I20260928 02:51:08.492950 24576 consensus_queue.cc:260] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.493054 24575 consensus_queue.cc:260] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.493479 24574 ts_tablet_manager.cc:1428] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:51:08.493788 24291 heartbeater.cc:503] Master 127.19.139.190:33511 was elected leader, sending a full tablet report...
I20260928 02:51:08.493901 24576 ts_tablet_manager.cc:1428] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:08.493929 24424 heartbeater.cc:503] Master 127.19.139.190:33511 was elected leader, sending a full tablet report...
I20260928 02:51:08.494215 24557 heartbeater.cc:503] Master 127.19.139.190:33511 was elected leader, sending a full tablet report...
I20260928 02:51:08.494640 24575 ts_tablet_manager.cc:1428] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:51:08.502238 24582 raft_consensus.cc:482] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:08.502432 24582 raft_consensus.cc:504] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.503952 24582 leader_election.cc:288] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d7477106d6b43018ce77428c5435adb (127.19.139.130:33571), 2705da223a084ed1acc62eae37677768 (127.19.139.131:36247)
I20260928 02:51:08.507380 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.503867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:48977 (local address 127.19.139.130:33571)
0928 02:51:08.504087 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.504093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.504183 (+ 90us) server_negotiation.cc:407] Connection header received
0928 02:51:08.504400 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.504404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.504458 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.504504 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:08.504898 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.505377 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.506460 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.506855 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.506969 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.507023 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.507184 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.507257 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 02:51:08.507380 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.504240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:56661 (local address 127.19.139.131:36247)
0928 02:51:08.504381 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.504385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.504398 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:08.504489 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.504492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.504532 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.504600 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:08.504859 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.505371 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.506452 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.506850 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.506954 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.507020 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.507166 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.507249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:51:08.507401 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.503746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:36247 (local address 127.19.139.129:56661)
0928 02:51:08.504292 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.504306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.504389 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.504611 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.504613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.504627 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:08.504756 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.504760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.505509 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.505512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.506341 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.506350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.506957 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.506961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.507052 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.507078 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.507135 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.507213 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":4,"spinlock_wait_cycles":27520,"thread_start_us":83,"threads_started":1}
I20260928 02:51:08.507405 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.503725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33571 (local address 127.19.139.129:48977)
0928 02:51:08.504083 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.504097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:08.504269 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:08.504641 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:08.504643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:08.504673 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:08.504787 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.504792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.505478 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.505481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.506342 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:08.506351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.506971 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.506975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:08.507077 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.507094 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:51:08.507163 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:51:08.507247 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":50,"thread_start_us":128,"threads_started":1}
I20260928 02:51:08.507972 24511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73de866e581f4499bd53b547760fb894" candidate_uuid: "573f11efdc3e4aeda6a882491452af19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2705da223a084ed1acc62eae37677768" is_pre_election: true
I20260928 02:51:08.508141 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73de866e581f4499bd53b547760fb894" candidate_uuid: "573f11efdc3e4aeda6a882491452af19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7477106d6b43018ce77428c5435adb" is_pre_election: true
I20260928 02:51:08.508404 24378 raft_consensus.cc:2454] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573f11efdc3e4aeda6a882491452af19 in term 0.
I20260928 02:51:08.508379 24511 raft_consensus.cc:2454] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573f11efdc3e4aeda6a882491452af19 in term 0.
I20260928 02:51:08.508901 24181 leader_election.cc:302] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7477106d6b43018ce77428c5435adb, 573f11efdc3e4aeda6a882491452af19; no voters:
I20260928 02:51:08.509192 24582 raft_consensus.cc:2781] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:08.509271 24582 raft_consensus.cc:482] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:08.509320 24582 raft_consensus.cc:3037] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:08.510353 24582 raft_consensus.cc:504] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.510727 24582 leader_election.cc:288] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [CANDIDATE]: Term 1 election: Requested vote from peers 1d7477106d6b43018ce77428c5435adb (127.19.139.130:33571), 2705da223a084ed1acc62eae37677768 (127.19.139.131:36247)
I20260928 02:51:08.511097 24511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73de866e581f4499bd53b547760fb894" candidate_uuid: "573f11efdc3e4aeda6a882491452af19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2705da223a084ed1acc62eae37677768"
I20260928 02:51:08.511097 24378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73de866e581f4499bd53b547760fb894" candidate_uuid: "573f11efdc3e4aeda6a882491452af19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7477106d6b43018ce77428c5435adb"
I20260928 02:51:08.511228 24378 raft_consensus.cc:3037] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:08.511229 24511 raft_consensus.cc:3037] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:08.512595 24378 raft_consensus.cc:2454] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573f11efdc3e4aeda6a882491452af19 in term 1.
I20260928 02:51:08.512595 24511 raft_consensus.cc:2454] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573f11efdc3e4aeda6a882491452af19 in term 1.
I20260928 02:51:08.512928 24181 leader_election.cc:302] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d7477106d6b43018ce77428c5435adb, 573f11efdc3e4aeda6a882491452af19; no voters:
I20260928 02:51:08.513105 24582 raft_consensus.cc:2781] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:08.513360 24582 raft_consensus.cc:686] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 1 LEADER]: Becoming Leader. State: Replica: 573f11efdc3e4aeda6a882491452af19, State: Running, Role: LEADER
I20260928 02:51:08.513871 24582 consensus_queue.cc:237] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } }
I20260928 02:51:08.517517 24100 catalog_manager.cc:5495] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 573f11efdc3e4aeda6a882491452af19 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "573f11efdc3e4aeda6a882491452af19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573f11efdc3e4aeda6a882491452af19" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 41729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 } health_report { overall_health: UNKNOWN } } }
I20260928 02:51:08.535640 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.532127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.19.139.129:41729)
0928 02:51:08.532273 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.532277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.532291 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:08.532393 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.532395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.532456 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.532546 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:08.532801 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.533330 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.534191 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:08.534403 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:08.534493 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:08.534818 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:08.535253 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:08.535363 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.535434 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.535497 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:51:08.541926 24511 raft_consensus.cc:1261] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 1 FOLLOWER]: Refusing update from remote peer 573f11efdc3e4aeda6a882491452af19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:08.541926 24378 raft_consensus.cc:1261] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 1 FOLLOWER]: Refusing update from remote peer 573f11efdc3e4aeda6a882491452af19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:08.542706 24582 consensus_queue.cc:1035] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7477106d6b43018ce77428c5435adb" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:08.542938 24585 consensus_queue.cc:1035] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2705da223a084ed1acc62eae37677768" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:08.552994 24591 mvcc.cc:205] Tried to move back new op lower bound from 7334149605539778560 to 7334149605436350464. Current Snapshot: MvccSnapshot[applied={T|T < 7334149605539778560}]
I20260928 02:51:08.560825 24100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:08.561004 24100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:08.563571 24100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4fe3808505e4f23baf571dde468f766: Sending DeleteTablet for 3 replicas of tablet 73de866e581f4499bd53b547760fb894
I20260928 02:51:08.564445 24225 tablet_service.cc:1514] Processing DeleteTablet for tablet 73de866e581f4499bd53b547760fb894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38978
I20260928 02:51:08.564558 24358 tablet_service.cc:1514] Processing DeleteTablet for tablet 73de866e581f4499bd53b547760fb894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:39492
I20260928 02:51:08.564580 24491 tablet_service.cc:1514] Processing DeleteTablet for tablet 73de866e581f4499bd53b547760fb894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:53412
I20260928 02:51:08.565109 24597 tablet_replica.cc:330] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: stopping tablet replica
I20260928 02:51:08.565183 24598 tablet_replica.cc:330] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19: stopping tablet replica
I20260928 02:51:08.565543 24598 raft_consensus.cc:2229] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:08.565546 24597 raft_consensus.cc:2229] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:08.565732 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24163
I20260928 02:51:08.565883 24597 raft_consensus.cc:2258] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:08.566035 24598 raft_consensus.cc:2258] T 73de866e581f4499bd53b547760fb894 P 573f11efdc3e4aeda6a882491452af19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:08.566399 24599 tablet_replica.cc:330] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb: stopping tablet replica
I20260928 02:51:08.566676 24599 raft_consensus.cc:2229] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:08.566951 24599 raft_consensus.cc:2258] T 73de866e581f4499bd53b547760fb894 P 1d7477106d6b43018ce77428c5435adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:08.568714 24597 ts_tablet_manager.cc:1905] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:08.571069 24597 ts_tablet_manager.cc:1918] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:08.571372 24597 log.cc:1196] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/73de866e581f4499bd53b547760fb894
I20260928 02:51:08.571558 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24296
I20260928 02:51:08.571798 24597 ts_tablet_manager.cc:1939] T 73de866e581f4499bd53b547760fb894 P 2705da223a084ed1acc62eae37677768: Deleting consensus metadata
I20260928 02:51:08.573567 24085 catalog_manager.cc:4841] TS 2705da223a084ed1acc62eae37677768 (127.19.139.131:36247): tablet 73de866e581f4499bd53b547760fb894 (table test-table [id=023db72b6ae040b58af1eff313ad5466]) successfully deleted
W20260928 02:51:08.573853 24085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:41729 (error 108)
I20260928 02:51:08.574877 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.574601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:41729 (local address 127.0.0.1:38990)
0928 02:51:08.574733 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.574800 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:51:08.575048 24085 catalog_manager.cc:4584] TS 573f11efdc3e4aeda6a882491452af19 (127.19.139.129:41729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73de866e581f4499bd53b547760fb894: Network error: Client connection negotiation failed: client connection to 127.19.139.129:41729: connect: Connection refused (error 111)
I20260928 02:51:08.577410 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24429
I20260928 02:51:08.578274 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.577512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33571 (local address 127.0.0.1:39498)
0928 02:51:08.578130 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:08.578187 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:33571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538}
I20260928 02:51:08.582320 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24070
2026-09-28T02:51:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2177 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:08Z Disabled control of system clock
I20260928 02:51:08.610450 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:38209
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:51:08.725502 24604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:08.725785 24604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:08.725823 24604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:08.729914 24604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:08.729991 24604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:08.730006 24604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:08.730022 24604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:08.730036 24604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:08.735111 24604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:38209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:08.736289 24604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:08.737681 24604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:08.746196 24610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:08.746206 24609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:08.746215 24612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:08.746548 24604 server_base.cc:1034] running on GCE node
I20260928 02:51:08.747181 24604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:08.748389 24604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:08.749567 24604 hybrid_clock.cc:648] HybridClock initialized: now 1790563868749553 us; error 30 us; skew 500 ppm
I20260928 02:51:08.751413 24604 webserver.cc:460] Webserver started at http://127.19.139.190:44185/ using document root <none> and password file <none>
I20260928 02:51:08.752015 24604 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:08.752086 24604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:08.752336 24604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:08.754483 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "6cca5b4e64e3469caad2af134939df43"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "cf79583b7a481025d4cf33b965fb966a"
server_key_iv: "f31925452857605437ae3d287016016b"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.755050 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "6cca5b4e64e3469caad2af134939df43"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "cf79583b7a481025d4cf33b965fb966a"
server_key_iv: "f31925452857605437ae3d287016016b"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.758994 24604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:08.761663 24618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.763109 24604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:51:08.763229 24604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "6cca5b4e64e3469caad2af134939df43"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "cf79583b7a481025d4cf33b965fb966a"
server_key_iv: "f31925452857605437ae3d287016016b"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.763365 24604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:08.768105 24604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:08.768817 24604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:08.769001 24604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:08.776114 24604 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:38209
I20260928 02:51:08.776116 24670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:38209 every 8 connection(s)
I20260928 02:51:08.777194 24604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:08.780550 24671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:08.786126 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24604
I20260928 02:51:08.786224 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:08.786356 24671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43: Bootstrap starting.
I20260928 02:51:08.786469 20014 external_mini_cluster.cc:1428] Setting key e553721150623a0ffee519934fd1bc40
I20260928 02:51:08.789028 24671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:08.789769 24671 log.cc:824] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:08.791806 24671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43: No bootstrap required, opened a new log
I20260928 02:51:08.792146 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:08.787814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.19.139.190:38209)
0928 02:51:08.788579 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:08.788596 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:08.788634 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:51:08.789351 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:08.789373 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:08.789755 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:08.790070 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:08.790815 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:08.790819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:08.790840 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:08.790850 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:08.790905 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:08.790922 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:08.791022 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:08.791167 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:51:08.791404 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":147,"threads_started":1}
I20260928 02:51:08.795789 24671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b4e64e3469caad2af134939df43" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38209 } }
I20260928 02:51:08.796044 24671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:08.796094 24671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cca5b4e64e3469caad2af134939df43, State: Initialized, Role: FOLLOWER
I20260928 02:51:08.796514 24671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b4e64e3469caad2af134939df43" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38209 } }
I20260928 02:51:08.796636 24671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:08.796695 24671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:08.796787 24671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:08.797704 24671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b4e64e3469caad2af134939df43" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38209 } }
I20260928 02:51:08.798053 24671 leader_election.cc:302] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cca5b4e64e3469caad2af134939df43; no voters:
I20260928 02:51:08.798386 24671 leader_election.cc:288] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:08.798498 24676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:08.798707 24676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [term 1 LEADER]: Becoming Leader. State: Replica: 6cca5b4e64e3469caad2af134939df43, State: Running, Role: LEADER
I20260928 02:51:08.799042 24676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b4e64e3469caad2af134939df43" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38209 } }
I20260928 02:51:08.799588 24671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:08.800622 24678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cca5b4e64e3469caad2af134939df43. Latest consensus state: current_term: 1 leader_uuid: "6cca5b4e64e3469caad2af134939df43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b4e64e3469caad2af134939df43" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38209 } } }
I20260928 02:51:08.800869 24678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:08.801286 24685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:08.801362 24677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cca5b4e64e3469caad2af134939df43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b4e64e3469caad2af134939df43" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 38209 } } }
I20260928 02:51:08.801482 24677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:08.804508 24685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:08.810046 24685 catalog_manager.cc:1347] Generated new cluster ID: 29fdcbe1f2ac43d99f7b51f9fb6db5a5
I20260928 02:51:08.810137 24685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:08.826798 24685 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:08.827145 24685 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:08.832815 24685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43: Generated new TSK 0
I20260928 02:51:08.833719 24685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:08.846364 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38209
--builtin_ntp_servers=127.19.139.148:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:51:08.959784 24695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:08.960086 24695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:08.960127 24695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:08.964365 24695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:08.964439 24695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:08.964498 24695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:08.969355 24695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:08.970436 24695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:08.971840 24695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:08.981775 24700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:08.981777 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:08.981976 24695 server_base.cc:1034] running on GCE node
W20260928 02:51:08.982162 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:08.982604 24695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:08.983323 24695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:08.984514 24695 hybrid_clock.cc:648] HybridClock initialized: now 1790563868984478 us; error 48 us; skew 500 ppm
I20260928 02:51:08.986529 24695 webserver.cc:460] Webserver started at http://127.19.139.129:34845/ using document root <none> and password file <none>
I20260928 02:51:08.987118 24695 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:08.987183 24695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:08.987442 24695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:08.989692 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "d33bee839f901221fefb0c5d8f4a0e24"
server_key_iv: "a22a287b0d3ee123308cef67a36868e4"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.990267 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "d33bee839f901221fefb0c5d8f4a0e24"
server_key_iv: "a22a287b0d3ee123308cef67a36868e4"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.994197 24695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:08.997072 24709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:08.998490 24695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:08.998620 24695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f"
format_stamp: "Formatted at 2026-09-28 02:51:08 on dist-test-slave-5vc0"
server_key: "d33bee839f901221fefb0c5d8f4a0e24"
server_key_iv: "a22a287b0d3ee123308cef67a36868e4"
server_key_version: "encryptionkey@0"
I20260928 02:51:08.998739 24695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:09.004050 24695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:09.004724 24695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:09.004885 24695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:09.005419 24695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:09.006336 24695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:09.006383 24695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.006433 24695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:09.006461 24695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.015345 24695 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:36419
I20260928 02:51:09.015367 24822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:36419 every 8 connection(s)
I20260928 02:51:09.016433 24695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:09.022639 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24695
I20260928 02:51:09.022753 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:09.023005 20014 external_mini_cluster.cc:1428] Setting key f911c4a9b5ba380bd4d12677a560240e
I20260928 02:51:09.025099 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38209
--builtin_ntp_servers=127.19.139.148:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:51:09.028332 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.018583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:60279 (local address 127.19.139.190:38209)
0928 02:51:09.018750 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.018765 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.019957 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 02:51:09.021419 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.021423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.021472 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.021551 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.027817 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.027820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.027823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.027837 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.027861 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.027870 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.027956 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.028118 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.028190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:51:09.029357 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.019057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:38209 (local address 127.19.139.129:60279)
0928 02:51:09.019790 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.019836 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.021127 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.021742 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.021750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.021853 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.027458 (+ 5605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.027461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.027522 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.027625 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.027966 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.027978 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.027986 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.028215 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.028534 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260928 02:51:09.030867 24823 heartbeater.cc:347] Connected to a master server at 127.19.139.190:38209
I20260928 02:51:09.031116 24823 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:09.031550 24823 heartbeater.cc:511] Master 127.19.139.190:38209 requested a full tablet report, sending...
I20260928 02:51:09.033470 24635 ts_manager.cc:194] Registered new tserver with Master: 06b9bb9cf4444dae8c7f4ff88f5b743f (127.19.139.129:36419)
W20260928 02:51:09.137317 24826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:09.137620 24826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:09.137661 24826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:09.141788 24826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:09.141851 24826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:09.141911 24826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:09.146878 24826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:09.147976 24826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:09.149405 24826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:51:09.159415 24826 server_base.cc:1034] running on GCE node
W20260928 02:51:09.159371 24832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:09.159329 24834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:09.159348 24831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:09.160080 24826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:09.160823 24826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:09.162021 24826 hybrid_clock.cc:648] HybridClock initialized: now 1790563869162001 us; error 37 us; skew 500 ppm
I20260928 02:51:09.164434 24826 webserver.cc:460] Webserver started at http://127.19.139.130:38141/ using document root <none> and password file <none>
I20260928 02:51:09.165091 24826 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:09.165164 24826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:09.165376 24826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:09.167735 24826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "0726228df47741e99031ead7b6647f8d"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "d54b299120e59a79c38c5c58663cdd07"
server_key_iv: "bb03a02c96e9d8956de2817a11eff407"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.168438 24826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "0726228df47741e99031ead7b6647f8d"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "d54b299120e59a79c38c5c58663cdd07"
server_key_iv: "bb03a02c96e9d8956de2817a11eff407"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.172883 24826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:51:09.176002 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.177640 24826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:09.177803 24826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "0726228df47741e99031ead7b6647f8d"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "d54b299120e59a79c38c5c58663cdd07"
server_key_iv: "bb03a02c96e9d8956de2817a11eff407"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.177963 24826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:09.183025 24826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:09.183712 24826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:09.183866 24826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:09.184482 24826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:09.185454 24826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:09.185499 24826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.185537 24826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:09.185557 24826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.194159 24826 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:39905
I20260928 02:51:09.194192 24953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:39905 every 8 connection(s)
I20260928 02:51:09.195111 24826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:09.201586 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24826
I20260928 02:51:09.201720 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:09.201807 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.197054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:45315 (local address 127.19.139.190:38209)
0928 02:51:09.197226 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.197229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.198361 (+ 1132us) server_negotiation.cc:407] Connection header received
0928 02:51:09.199867 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.199871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.199913 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.199988 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.201296 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.201300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.201303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.201317 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.201339 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.201348 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.201444 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.201598 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.201638 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:51:09.202055 20014 external_mini_cluster.cc:1428] Setting key ff6103bb0acfb053e9a676724c16f72d
I20260928 02:51:09.203054 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.197481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:38209 (local address 127.19.139.130:45315)
0928 02:51:09.198192 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.198238 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.199515 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.200186 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.200194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.200348 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.201003 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.201005 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.201051 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.201127 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.201443 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.201452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.201459 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.201650 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.202017 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260928 02:51:09.204231 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38209
--builtin_ntp_servers=127.19.139.148:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:51:09.204306 24954 heartbeater.cc:347] Connected to a master server at 127.19.139.190:38209
I20260928 02:51:09.204533 24954 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:09.205045 24954 heartbeater.cc:511] Master 127.19.139.190:38209 requested a full tablet report, sending...
I20260928 02:51:09.206365 24635 ts_manager.cc:194] Registered new tserver with Master: 0726228df47741e99031ead7b6647f8d (127.19.139.130:39905)
W20260928 02:51:09.318207 24957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:09.318545 24957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:09.318588 24957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:09.322723 24957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:09.322794 24957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:09.322854 24957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:09.327765 24957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:09.328980 24957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:09.330384 24957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:09.340619 24962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:09.340598 24963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:09.340651 24957 server_base.cc:1034] running on GCE node
W20260928 02:51:09.340598 24965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:09.341403 24957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:09.342229 24957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:09.343431 24957 hybrid_clock.cc:648] HybridClock initialized: now 1790563869343416 us; error 36 us; skew 500 ppm
I20260928 02:51:09.345418 24957 webserver.cc:460] Webserver started at http://127.19.139.131:36263/ using document root <none> and password file <none>
I20260928 02:51:09.346011 24957 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:09.346077 24957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:09.346259 24957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:09.348701 24957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "a5657e4fb31b4faf92871df0b081833b"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "4b1815b67fe741287e08864c9dcd3b92"
server_key_iv: "bc3e124339c63623fd49a125897e0e04"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.349280 24957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "a5657e4fb31b4faf92871df0b081833b"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "4b1815b67fe741287e08864c9dcd3b92"
server_key_iv: "bc3e124339c63623fd49a125897e0e04"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.353837 24957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:09.356976 24971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.358325 24957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:09.358438 24957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "a5657e4fb31b4faf92871df0b081833b"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "4b1815b67fe741287e08864c9dcd3b92"
server_key_iv: "bc3e124339c63623fd49a125897e0e04"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.358546 24957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:09.363310 24957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:09.363914 24957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:09.364048 24957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:09.364598 24957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:09.365511 24957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:09.365558 24957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.365609 24957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:09.365635 24957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.374030 24957 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:38171
I20260928 02:51:09.374073 25084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:38171 every 8 connection(s)
I20260928 02:51:09.374979 24957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:09.381132 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 24957
I20260928 02:51:09.381253 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:09.381371 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.377014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:47127 (local address 127.19.139.190:38209)
0928 02:51:09.377151 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.377155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.378292 (+ 1137us) server_negotiation.cc:407] Connection header received
0928 02:51:09.379641 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.379645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.379703 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.379782 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.380902 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.380905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.380908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.380922 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.380946 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.380954 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.381035 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.381151 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.381197 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:51:09.381541 20014 external_mini_cluster.cc:1428] Setting key 61323f9c55cd6b025422ac66b7e711b8
I20260928 02:51:09.382396 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.377428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:38209 (local address 127.19.139.131:47127)
0928 02:51:09.378115 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.378172 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.379416 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.379963 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.379971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.380068 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.380641 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.380643 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.380686 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.380754 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.381022 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.381029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.381036 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.381191 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.381523 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":115,"threads_started":1}
I20260928 02:51:09.384035 25085 heartbeater.cc:347] Connected to a master server at 127.19.139.190:38209
I20260928 02:51:09.384284 25085 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:09.384703 25085 heartbeater.cc:511] Master 127.19.139.190:38209 requested a full tablet report, sending...
I20260928 02:51:09.385735 24634 ts_manager.cc:194] Registered new tserver with Master: a5657e4fb31b4faf92871df0b081833b (127.19.139.131:38171)
I20260928 02:51:09.395937 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:51:09.399681 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.397931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.19.139.190:38209)
0928 02:51:09.398070 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.398074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.398213 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:51:09.398395 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.398398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.398434 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.398497 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.399299 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.399303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.399306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.399317 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.399337 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.399346 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.399422 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.399493 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.399543 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:51:09.403069 24634 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:09.406172 24634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:09.419361 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.415665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:38171 (local address 127.0.0.1:58708)
0928 02:51:09.416643 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.416658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.416741 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.417451 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.417458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.417544 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.418435 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.418437 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.418480 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.418554 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.419064 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.419067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.419070 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.419120 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.419210 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":77,"threads_started":1}
I20260928 02:51:09.419386 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.416885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.19.139.130:39905)
0928 02:51:09.417462 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.417474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.417496 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:51:09.417541 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.417545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.417713 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.417856 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.418763 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.418770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.418784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.418797 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.418847 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.418884 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.419029 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.419133 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.419215 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":141,"threads_started":1}
I20260928 02:51:09.419762 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.416346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.19.139.129:36419)
0928 02:51:09.416880 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.416891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.416917 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:51:09.416976 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.416981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.417207 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.417354 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.418841 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.418846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.418855 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.418867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.418905 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.418941 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.419029 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.419240 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.419304 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":138,"threads_started":1}
I20260928 02:51:09.419831 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.416488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39905 (local address 127.0.0.1:45742)
0928 02:51:09.417320 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.417333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.417396 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.418191 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.418194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.418209 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.418594 (+ 385us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.418596 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.418607 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.418636 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.418988 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.418999 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.419005 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.419120 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.419197 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":87,"threads_started":1}
I20260928 02:51:09.420018 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.415780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.19.139.131:38171)
0928 02:51:09.416989 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.417000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.417024 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:51:09.417077 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.417082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.417273 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.417987 (+ 714us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.418899 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.418902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.418911 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.418918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.418949 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.418970 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.419756 (+ 786us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.419804 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.419877 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1104,"thread_start_us":125,"threads_started":1}
I20260928 02:51:09.421093 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.415777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:36419 (local address 127.0.0.1:36330)
0928 02:51:09.416394 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.416456 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.416569 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.417501 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.417504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.417544 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.418685 (+ 1141us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.418686 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.418696 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.418719 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.419072 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.419075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.419078 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.419148 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.420996 (+ 1848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":761,"thread_start_us":128,"threads_started":1}
I20260928 02:51:09.423209 24888 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf403b9e3c249878f1dac6535b8be23 (DEFAULT_TABLE table=test-table [id=be6471251614456cbb36e62b1859634c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:09.423621 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf403b9e3c249878f1dac6535b8be23 (DEFAULT_TABLE table=test-table [id=be6471251614456cbb36e62b1859634c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:09.423700 24757 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf403b9e3c249878f1dac6535b8be23 (DEFAULT_TABLE table=test-table [id=be6471251614456cbb36e62b1859634c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:09.424536 24888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf403b9e3c249878f1dac6535b8be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:09.424589 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf403b9e3c249878f1dac6535b8be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:09.424671 24757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf403b9e3c249878f1dac6535b8be23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:09.431043 25102 tablet_bootstrap.cc:492] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Bootstrap starting.
I20260928 02:51:09.432459 25104 tablet_bootstrap.cc:492] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Bootstrap starting.
I20260928 02:51:09.432889 25103 tablet_bootstrap.cc:492] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: Bootstrap starting.
I20260928 02:51:09.433840 25102 tablet_bootstrap.cc:654] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:09.434706 25103 tablet_bootstrap.cc:654] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:09.434760 25102 log.cc:824] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:09.435482 25103 log.cc:824] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:09.435992 25104 tablet_bootstrap.cc:654] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:09.436826 25104 log.cc:824] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:09.437191 25102 tablet_bootstrap.cc:492] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: No bootstrap required, opened a new log
I20260928 02:51:09.437229 25103 tablet_bootstrap.cc:492] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: No bootstrap required, opened a new log
I20260928 02:51:09.437412 25102 ts_tablet_manager.cc:1397] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:51:09.437413 25103 ts_tablet_manager.cc:1397] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:51:09.438861 25104 tablet_bootstrap.cc:492] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: No bootstrap required, opened a new log
I20260928 02:51:09.439059 25104 ts_tablet_manager.cc:1397] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:51:09.441216 25103 raft_consensus.cc:348] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.441502 25103 raft_consensus.cc:374] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:09.441576 25103 raft_consensus.cc:729] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b9bb9cf4444dae8c7f4ff88f5b743f, State: Initialized, Role: FOLLOWER
I20260928 02:51:09.442029 25103 consensus_queue.cc:260] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.441929 25102 raft_consensus.cc:348] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.442268 25102 raft_consensus.cc:374] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:09.442328 25102 raft_consensus.cc:729] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5657e4fb31b4faf92871df0b081833b, State: Initialized, Role: FOLLOWER
I20260928 02:51:09.442775 25102 consensus_queue.cc:260] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.443027 25103 ts_tablet_manager.cc:1428] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:51:09.443382 24823 heartbeater.cc:503] Master 127.19.139.190:38209 was elected leader, sending a full tablet report...
I20260928 02:51:09.443538 25104 raft_consensus.cc:348] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.443758 25102 ts_tablet_manager.cc:1428] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:51:09.443794 25104 raft_consensus.cc:374] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:09.443882 25104 raft_consensus.cc:729] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0726228df47741e99031ead7b6647f8d, State: Initialized, Role: FOLLOWER
I20260928 02:51:09.444109 25085 heartbeater.cc:503] Master 127.19.139.190:38209 was elected leader, sending a full tablet report...
I20260928 02:51:09.444344 25104 consensus_queue.cc:260] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.445286 25104 ts_tablet_manager.cc:1428] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:51:09.445775 24954 heartbeater.cc:503] Master 127.19.139.190:38209 was elected leader, sending a full tablet report...
W20260928 02:51:09.446002 24955 tablet.cc:2369] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:51:09.498242 25109 raft_consensus.cc:482] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:09.498466 25109 raft_consensus.cc:504] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.499673 25109 leader_election.cc:288] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06b9bb9cf4444dae8c7f4ff88f5b743f (127.19.139.129:36419), 0726228df47741e99031ead7b6647f8d (127.19.139.130:39905)
I20260928 02:51:09.501571 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.499917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:36419 (local address 127.19.139.131:33525)
0928 02:51:09.500045 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.500061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.500180 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.500528 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.500531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.500547 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.501053 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.501054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.501064 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.501091 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.501326 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.501329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.501332 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.501403 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.501438 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:51:09.501571 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.500020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:33525 (local address 127.19.139.129:36419)
0928 02:51:09.500144 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.500149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.500199 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:51:09.500281 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.500285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.500350 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.500424 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.501206 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.501210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.501213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.501224 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.501246 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.501253 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.501337 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.501402 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.501441 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:51:09.501833 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.500365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:58021 (local address 127.19.139.130:39905)
0928 02:51:09.500514 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.500518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.500649 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:51:09.500722 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.500725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.500761 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.500826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.501394 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.501398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.501400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.501413 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.501433 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.501442 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.501520 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.501667 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.501711 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:51:09.502141 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.500206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39905 (local address 127.19.139.131:58021)
0928 02:51:09.500556 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.500569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.500656 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.500858 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.500861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.500882 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:09.501242 (+ 360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:09.501244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:09.501253 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:09.501275 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:09.501543 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:09.501546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:09.501549 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:51:09.501987 (+ 438us) client_negotiation.cc:240] Negotiation successful
0928 02:51:09.502020 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260928 02:51:09.502245 24777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf403b9e3c249878f1dac6535b8be23" candidate_uuid: "a5657e4fb31b4faf92871df0b081833b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" is_pre_election: true
I20260928 02:51:09.502609 24777 raft_consensus.cc:2454] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5657e4fb31b4faf92871df0b081833b in term 0.
I20260928 02:51:09.502898 24908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf403b9e3c249878f1dac6535b8be23" candidate_uuid: "a5657e4fb31b4faf92871df0b081833b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0726228df47741e99031ead7b6647f8d" is_pre_election: true
I20260928 02:51:09.503155 24972 leader_election.cc:302] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b9bb9cf4444dae8c7f4ff88f5b743f, a5657e4fb31b4faf92871df0b081833b; no voters:
I20260928 02:51:09.503199 24908 raft_consensus.cc:2454] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5657e4fb31b4faf92871df0b081833b in term 0.
I20260928 02:51:09.503460 25109 raft_consensus.cc:2781] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:09.503556 25109 raft_consensus.cc:482] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:09.503639 25109 raft_consensus.cc:3037] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:09.504812 25109 raft_consensus.cc:504] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.505128 25109 leader_election.cc:288] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [CANDIDATE]: Term 1 election: Requested vote from peers 06b9bb9cf4444dae8c7f4ff88f5b743f (127.19.139.129:36419), 0726228df47741e99031ead7b6647f8d (127.19.139.130:39905)
I20260928 02:51:09.505493 24777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf403b9e3c249878f1dac6535b8be23" candidate_uuid: "a5657e4fb31b4faf92871df0b081833b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f"
I20260928 02:51:09.505589 24908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf403b9e3c249878f1dac6535b8be23" candidate_uuid: "a5657e4fb31b4faf92871df0b081833b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0726228df47741e99031ead7b6647f8d"
I20260928 02:51:09.505635 24777 raft_consensus.cc:3037] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:09.505707 24908 raft_consensus.cc:3037] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:09.506940 24777 raft_consensus.cc:2454] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5657e4fb31b4faf92871df0b081833b in term 1.
I20260928 02:51:09.506940 24908 raft_consensus.cc:2454] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5657e4fb31b4faf92871df0b081833b in term 1.
I20260928 02:51:09.507234 24972 leader_election.cc:302] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b9bb9cf4444dae8c7f4ff88f5b743f, a5657e4fb31b4faf92871df0b081833b; no voters:
I20260928 02:51:09.507507 25109 raft_consensus.cc:2781] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:09.507750 25109 raft_consensus.cc:686] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 1 LEADER]: Becoming Leader. State: Replica: a5657e4fb31b4faf92871df0b081833b, State: Running, Role: LEADER
I20260928 02:51:09.508092 25109 consensus_queue.cc:237] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } }
I20260928 02:51:09.511464 24633 catalog_manager.cc:5495] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5657e4fb31b4faf92871df0b081833b (127.19.139.131). New cstate: current_term: 1 leader_uuid: "a5657e4fb31b4faf92871df0b081833b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5657e4fb31b4faf92871df0b081833b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 } health_report { overall_health: UNKNOWN } } }
W20260928 02:51:09.517591 24824 tablet.cc:2369] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:51:09.543852 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.542405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.19.139.131:38171)
0928 02:51:09.542576 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.542580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.542594 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:09.542670 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.542672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.542730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.542808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.543470 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.543474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.543476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.543486 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.543505 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.543513 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.543586 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.543705 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.543738 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 02:51:09.550267 24908 raft_consensus.cc:1261] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 1 FOLLOWER]: Refusing update from remote peer a5657e4fb31b4faf92871df0b081833b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:09.550267 24777 raft_consensus.cc:1261] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 1 FOLLOWER]: Refusing update from remote peer a5657e4fb31b4faf92871df0b081833b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:09.550952 25112 consensus_queue.cc:1035] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0726228df47741e99031ead7b6647f8d" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:09.551134 25109 consensus_queue.cc:1035] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06b9bb9cf4444dae8c7f4ff88f5b743f" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:09.559238 25117 mvcc.cc:205] Tried to move back new op lower bound from 7334149609670144000 to 7334149609508003840. Current Snapshot: MvccSnapshot[applied={T|T < 7334149609670144000}]
I20260928 02:51:09.568099 24634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:09.568344 24634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:09.571200 24634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cca5b4e64e3469caad2af134939df43: Sending DeleteTablet for 3 replicas of tablet 1bf403b9e3c249878f1dac6535b8be23
I20260928 02:51:09.572239 24757 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf403b9e3c249878f1dac6535b8be23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:09 UTC) from {username='slave'} at 127.0.0.1:36330
I20260928 02:51:09.572183 25019 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf403b9e3c249878f1dac6535b8be23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:09 UTC) from {username='slave'} at 127.0.0.1:58708
I20260928 02:51:09.572682 24888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf403b9e3c249878f1dac6535b8be23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:09 UTC) from {username='slave'} at 127.0.0.1:45742
I20260928 02:51:09.572873 25125 tablet_replica.cc:330] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f: stopping tablet replica
I20260928 02:51:09.573232 25127 tablet_replica.cc:330] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: stopping tablet replica
I20260928 02:51:09.573251 25125 raft_consensus.cc:2229] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:09.573535 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24695
I20260928 02:51:09.573580 25126 tablet_replica.cc:330] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: stopping tablet replica
I20260928 02:51:09.573734 25125 raft_consensus.cc:2258] T 1bf403b9e3c249878f1dac6535b8be23 P 06b9bb9cf4444dae8c7f4ff88f5b743f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:09.573859 25126 raft_consensus.cc:2229] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:09.574276 25126 raft_consensus.cc:2258] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:09.574285 25127 raft_consensus.cc:2229] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:09.574560 25127 raft_consensus.cc:2258] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:09.575905 25126 ts_tablet_manager.cc:1905] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:09.576819 25127 ts_tablet_manager.cc:1905] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:09.578855 25126 ts_tablet_manager.cc:1918] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:09.579043 25126 log.cc:1196] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/1bf403b9e3c249878f1dac6535b8be23
I20260928 02:51:09.579438 25126 ts_tablet_manager.cc:1939] T 1bf403b9e3c249878f1dac6535b8be23 P a5657e4fb31b4faf92871df0b081833b: Deleting consensus metadata
I20260928 02:51:09.579522 25127 ts_tablet_manager.cc:1918] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:09.579664 25127 log.cc:1196] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/1bf403b9e3c249878f1dac6535b8be23
I20260928 02:51:09.579990 25127 ts_tablet_manager.cc:1939] T 1bf403b9e3c249878f1dac6535b8be23 P 0726228df47741e99031ead7b6647f8d: Deleting consensus metadata
I20260928 02:51:09.580868 24620 catalog_manager.cc:4841] TS a5657e4fb31b4faf92871df0b081833b (127.19.139.131:38171): tablet 1bf403b9e3c249878f1dac6535b8be23 (table test-table [id=be6471251614456cbb36e62b1859634c]) successfully deleted
W20260928 02:51:09.581322 24619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:36419 (error 108)
I20260928 02:51:09.581516 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24826
I20260928 02:51:09.581799 24619 catalog_manager.cc:4841] TS 0726228df47741e99031ead7b6647f8d (127.19.139.130:39905): tablet 1bf403b9e3c249878f1dac6535b8be23 (table test-table [id=be6471251614456cbb36e62b1859634c]) successfully deleted
I20260928 02:51:09.583151 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.582757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:36419 (local address 127.0.0.1:36332)
0928 02:51:09.582939 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.583062 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 02:51:09.583392 24619 catalog_manager.cc:4584] TS 06b9bb9cf4444dae8c7f4ff88f5b743f (127.19.139.129:36419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bf403b9e3c249878f1dac6535b8be23: Network error: Client connection negotiation failed: client connection to 127.19.139.129:36419: connect: Connection refused (error 111)
I20260928 02:51:09.587832 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24957
I20260928 02:51:09.592458 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 24604
2026-09-28T02:51:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1009 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:51:09.612600 20014 mini_oidc.cc:152] Starting JWKS server
I20260928 02:51:09.614156 20014 webserver.cc:460] Webserver started at https://127.0.0.1:35505/ using document root <none> and password file <none>
I20260928 02:51:09.614301 20014 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:51:09.614529 20014 webserver.cc:460] Webserver started at http://127.0.0.1:35973/ using document root <none> and password file <none>
2026-09-28T02:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:09Z Disabled control of system clock
I20260928 02:51:09.622975 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37617
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37617
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/testchainca.cert with env {}
W20260928 02:51:09.733081 25134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:09.733378 25134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:09.733419 25134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:09.737535 25134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:09.737604 25134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:09.737618 25134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:09.737633 25134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:09.737648 25134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:51:09.737663 25134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:51:09.737674 25134 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35505/jwks.json
I20260928 02:51:09.743254 25134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:09.744407 25134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:09.745458 25134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:09.754338 25139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:09.754487 25142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:09.754355 25140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:09.754421 25134 server_base.cc:1034] running on GCE node
I20260928 02:51:09.755291 25134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:09.756538 25134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:09.757750 25134 hybrid_clock.cc:648] HybridClock initialized: now 1790563869757725 us; error 40 us; skew 500 ppm
I20260928 02:51:09.759778 25134 webserver.cc:460] Webserver started at http://127.19.139.190:37165/ using document root <none> and password file <none>
I20260928 02:51:09.760434 25134 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:09.760507 25134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:09.760718 25134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:09.762929 25134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "6bdcd38b403f4056ac0dad1d386fb9ef"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "6386d4a1c9bc8ae6eff9c76d7b146522"
server_key_iv: "400485a41409277502cb386a2e3bc024"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.763537 25134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "6bdcd38b403f4056ac0dad1d386fb9ef"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "6386d4a1c9bc8ae6eff9c76d7b146522"
server_key_iv: "400485a41409277502cb386a2e3bc024"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.767544 25134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:09.770390 25148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:09.771816 25134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:51:09.771950 25134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "6bdcd38b403f4056ac0dad1d386fb9ef"
format_stamp: "Formatted at 2026-09-28 02:51:09 on dist-test-slave-5vc0"
server_key: "6386d4a1c9bc8ae6eff9c76d7b146522"
server_key_iv: "400485a41409277502cb386a2e3bc024"
server_key_version: "encryptionkey@0"
I20260928 02:51:09.772070 25134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:09.787320 25134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:09.788075 25134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:09.788293 25134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:09.795331 25134 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:37617
I20260928 02:51:09.795378 25200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:37617 every 8 connection(s)
I20260928 02:51:09.796367 25134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:09.798858 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25134
I20260928 02:51:09.798987 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:09.799270 20014 external_mini_cluster.cc:1428] Setting key 49acfe8be396a0ccc5d3ed47513e4f08
I20260928 02:51:09.799904 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:09.806831 25201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef: Bootstrap starting.
I20260928 02:51:09.808604 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.800654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.19.139.190:37617)
0928 02:51:09.801267 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.801283 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.801319 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:51:09.802004 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.802023 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.802432 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.802737 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:09.803100 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:09.804078 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:09.805254 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:09.805748 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:09.805862 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:09.806716 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:09.806722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:09.806734 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:09.807161 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:09.807220 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:09.807241 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:09.807316 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:09.807513 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:51:09.807794 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260928 02:51:09.809266 25201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:09.810005 25201 log.cc:824] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:09.811580 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:51:09.812330 25201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef: No bootstrap required, opened a new log
I20260928 02:51:09.816047 25201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37617 } }
I20260928 02:51:09.816358 25201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:09.816408 25201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdcd38b403f4056ac0dad1d386fb9ef, State: Initialized, Role: FOLLOWER
I20260928 02:51:09.816836 25201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37617 } }
I20260928 02:51:09.816967 25201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:09.817009 25201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:09.817088 25201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:09.818096 25201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37617 } }
I20260928 02:51:09.818440 25201 leader_election.cc:302] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bdcd38b403f4056ac0dad1d386fb9ef; no voters:
I20260928 02:51:09.818779 25201 leader_election.cc:288] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:09.818993 25206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:09.819299 25206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [term 1 LEADER]: Becoming Leader. State: Replica: 6bdcd38b403f4056ac0dad1d386fb9ef, State: Running, Role: LEADER
I20260928 02:51:09.819694 25206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37617 } }
I20260928 02:51:09.820219 25201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:09.821360 25207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37617 } } }
I20260928 02:51:09.821482 25207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:09.821880 25215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:09.821841 25208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bdcd38b403f4056ac0dad1d386fb9ef. Latest consensus state: current_term: 1 leader_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdcd38b403f4056ac0dad1d386fb9ef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37617 } } }
I20260928 02:51:09.821975 25208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:09.825304 25215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:09.831254 25215 catalog_manager.cc:1347] Generated new cluster ID: e6fd1d90ffc74c91863cbb84fef41654
I20260928 02:51:09.831342 25215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:09.850077 25215 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:09.851408 25215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:09.860770 25215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bdcd38b403f4056ac0dad1d386fb9ef: Generated new TSK 0
I20260928 02:51:09.861757 25215 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:51:09.895116 25231 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:51:09.895270 25231 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:51:09.895797 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:09.874699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.19.139.190:37617)
0928 02:51:09.874848 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:09.874852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:09.875056 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:51:09.875255 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:09.875258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:09.875312 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:09.875404 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:51:09.875698 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:09.876430 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:09.877701 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:09.877956 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:09.878035 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:09.878430 (+ 395us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:51:09.895397 (+ 16967us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:51:09.895665 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 02:51:09.895884 25204 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:51:09.897426 25230 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:09.874563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37617 (local address 127.0.0.1:58218)
0928 02:51:09.874920 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:09.874936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:09.875083 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:09.875435 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:09.875439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:09.875455 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:51:09.875584 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:09.875590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:09.876540 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:09.876543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:09.877572 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:09.877584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:09.878062 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:09.878066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:09.878157 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:09.878217 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:51:09.895682 (+ 17465us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:51:09.895995 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.139.190:37617: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.139.190:37617: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35505/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:51:09.898224 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25134
2026-09-28T02:51:09Z chronyd exiting
I20260928 02:51:09.934967 20014 test_util.cc:175] -----------------------------------------------
I20260928 02:51:09.935057 20014 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563845681472-20014-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:51:09.936142 20014 mini_oidc.cc:152] Starting JWKS server
I20260928 02:51:09.937070 20014 webserver.cc:460] Webserver started at https://127.0.0.1:44463/ using document root <none> and password file <none>
I20260928 02:51:09.937170 20014 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:51:09.937417 20014 webserver.cc:460] Webserver started at http://127.0.0.1:43561/ using document root <none> and password file <none>
2026-09-28T02:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:09Z Disabled control of system clock
I20260928 02:51:09.946273 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40877
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:40877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/testchainca.cert with env {}
W20260928 02:51:10.057444 25237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:10.057749 25237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:10.057789 25237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:10.061990 25237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:10.062064 25237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:10.062078 25237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:10.062094 25237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:10.062109 25237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:51:10.062124 25237 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:51:10.062136 25237 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44463/jwks.json
I20260928 02:51:10.067787 25237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:40877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:10.069058 25237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:10.070104 25237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:10.078660 25243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:10.078683 25242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:10.078660 25245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:10.078708 25237 server_base.cc:1034] running on GCE node
I20260928 02:51:10.079564 25237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:10.080766 25237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:10.082003 25237 hybrid_clock.cc:648] HybridClock initialized: now 1790563870081985 us; error 38 us; skew 500 ppm
I20260928 02:51:10.084051 25237 webserver.cc:460] Webserver started at http://127.19.139.190:44155/ using document root <none> and password file <none>
I20260928 02:51:10.084721 25237 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:10.084797 25237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:10.084990 25237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:10.087090 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "80c841316b6f6ba9fb7ca830746fafdd"
server_key_iv: "ae9815e4af006d2d761bccafb4dddcda"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.087653 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "80c841316b6f6ba9fb7ca830746fafdd"
server_key_iv: "ae9815e4af006d2d761bccafb4dddcda"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.091549 25237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:10.094172 25251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:10.095538 25237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:10.095690 25237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "80c841316b6f6ba9fb7ca830746fafdd"
server_key_iv: "ae9815e4af006d2d761bccafb4dddcda"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.095810 25237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:10.113121 25237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:10.113879 25237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:10.114076 25237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:10.120719 25237 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:40877
I20260928 02:51:10.120747 25303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:40877 every 8 connection(s)
I20260928 02:51:10.121680 25237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:10.122056 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25237
I20260928 02:51:10.122182 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:10.122530 20014 external_mini_cluster.cc:1428] Setting key aae26b1b41454183d156821a5e4585f7
I20260928 02:51:10.125310 25304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:10.130923 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10: Bootstrap starting.
I20260928 02:51:10.131665 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:10.123883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.19.139.190:40877)
0928 02:51:10.124594 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:10.124612 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:10.124649 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:51:10.125305 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:10.125322 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:10.125757 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:10.126081 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:10.126520 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.127498 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.128434 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.128916 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.129060 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.129911 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:10.129916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:10.129929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:10.130323 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:10.130398 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:10.130419 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:10.130503 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:10.130641 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:51:10.130907 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260928 02:51:10.133353 25304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:10.134191 25304 log.cc:824] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:10.134598 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:51:10.136390 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10: No bootstrap required, opened a new log
I20260928 02:51:10.140017 25304 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40877 } }
I20260928 02:51:10.140326 25304 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:10.140372 25304 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a79e8d715e4de49f0cfe2b8bf7bf10, State: Initialized, Role: FOLLOWER
I20260928 02:51:10.140817 25304 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40877 } }
I20260928 02:51:10.140952 25304 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:10.141021 25304 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:10.141113 25304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:10.142261 25304 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40877 } }
I20260928 02:51:10.142637 25304 leader_election.cc:302] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6a79e8d715e4de49f0cfe2b8bf7bf10; no voters:
I20260928 02:51:10.142959 25304 leader_election.cc:288] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:10.143108 25309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:10.143332 25309 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [term 1 LEADER]: Becoming Leader. State: Replica: b6a79e8d715e4de49f0cfe2b8bf7bf10, State: Running, Role: LEADER
I20260928 02:51:10.143646 25309 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40877 } }
I20260928 02:51:10.144189 25304 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:10.145229 25311 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a79e8d715e4de49f0cfe2b8bf7bf10. Latest consensus state: current_term: 1 leader_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40877 } } }
I20260928 02:51:10.145313 25310 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a79e8d715e4de49f0cfe2b8bf7bf10" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40877 } } }
I20260928 02:51:10.145469 25310 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:10.145428 25311 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:10.145888 25319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:10.148939 25319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:10.154321 25319 catalog_manager.cc:1347] Generated new cluster ID: b3d013c4e08243239635dd83595ed133
I20260928 02:51:10.154399 25319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:10.166901 25319 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:10.167860 25319 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:10.177254 25319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6a79e8d715e4de49f0cfe2b8bf7bf10: Generated new TSK 0
I20260928 02:51:10.178190 25319 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:51:10.211732 25334 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:51:10.211813 25334 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:51:10.212702 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:10.196716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.19.139.190:40877)
0928 02:51:10.196852 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:10.196857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:10.197001 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:51:10.197147 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:10.197150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:10.197206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:10.197319 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:51:10.197499 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.198317 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.199215 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.199546 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.199643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.199893 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:51:10.212234 (+ 12341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:51:10.212542 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:51:10.212780 25307 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:51:10.212783 25333 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:10.196612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40877 (local address 127.0.0.1:48618)
0928 02:51:10.196918 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:10.196930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:10.197033 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:10.197311 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:10.197313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:10.197319 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:51:10.197414 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:10.197418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.198433 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.198436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:10.199117 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:10.199126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.199642 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.199644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:10.199701 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.199726 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:51:10.212470 (+ 12744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:51:10.212632 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.139.190:40877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260928 02:51:10.213568 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25237
2026-09-28T02:51:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:51:10.251966 20014 mini_oidc.cc:152] Starting JWKS server
I20260928 02:51:10.252877 20014 webserver.cc:460] Webserver started at https://127.0.0.1:37103/ using document root <none> and password file <none>
I20260928 02:51:10.252969 20014 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:51:10.253185 20014 webserver.cc:460] Webserver started at http://127.0.0.1:46679/ using document root <none> and password file <none>
2026-09-28T02:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:10Z Disabled control of system clock
I20260928 02:51:10.261984 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40251
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:40251
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37103/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:51:10.371441 25340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:10.371738 25340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:10.371785 25340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:10.376327 25340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:10.376421 25340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:10.376436 25340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:10.376453 25340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:10.376488 25340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:51:10.376521 25340 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:51:10.376547 25340 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37103/jwks.json
I20260928 02:51:10.381748 25340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:40251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37103/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:10.382943 25340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:10.384413 25340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:10.392716 25346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:10.392710 25345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:10.392710 25348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:10.393146 25340 server_base.cc:1034] running on GCE node
I20260928 02:51:10.393780 25340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:10.395236 25340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:10.396445 25340 hybrid_clock.cc:648] HybridClock initialized: now 1790563870396428 us; error 38 us; skew 500 ppm
I20260928 02:51:10.398514 25340 webserver.cc:460] Webserver started at http://127.19.139.190:45893/ using document root <none> and password file <none>
I20260928 02:51:10.399097 25340 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:10.399170 25340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:10.399385 25340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:10.401573 25340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "c8b3748d5ab641558812b6840f26159b"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "4f6ebc4a80d64a72b2ba985f97c16553"
server_key_iv: "77d0bf042a15691d5cb40587b2ebf26c"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.402159 25340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "c8b3748d5ab641558812b6840f26159b"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "4f6ebc4a80d64a72b2ba985f97c16553"
server_key_iv: "77d0bf042a15691d5cb40587b2ebf26c"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.406054 25340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:10.408581 25354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:10.409824 25340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:10.409945 25340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "c8b3748d5ab641558812b6840f26159b"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "4f6ebc4a80d64a72b2ba985f97c16553"
server_key_iv: "77d0bf042a15691d5cb40587b2ebf26c"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.410060 25340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:10.440708 25340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:10.441452 25340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:10.441633 25340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:10.448280 25340 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:40251
I20260928 02:51:10.448335 25406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:40251 every 8 connection(s)
I20260928 02:51:10.449275 25340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:10.452491 25407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:10.458240 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25340
I20260928 02:51:10.458349 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:10.458418 25407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b: Bootstrap starting.
I20260928 02:51:10.458638 20014 external_mini_cluster.cc:1428] Setting key 65449660aafc60589890b275bdeb4f79
I20260928 02:51:10.460950 25407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:10.461700 25407 log.cc:824] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:10.463658 25407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b: No bootstrap required, opened a new log
I20260928 02:51:10.467430 25407 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b3748d5ab641558812b6840f26159b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40251 } }
I20260928 02:51:10.467686 25407 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:10.467746 25407 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b3748d5ab641558812b6840f26159b, State: Initialized, Role: FOLLOWER
I20260928 02:51:10.468119 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:10.460028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.19.139.190:40251)
0928 02:51:10.460710 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:10.460725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:10.460781 (+ 56us) server_negotiation.cc:407] Connection header received
0928 02:51:10.461455 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:10.461475 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:10.461931 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:10.462283 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:10.462694 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.463670 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.464600 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.465072 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.465215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.466178 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:10.466183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:10.466206 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:10.466635 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:10.466696 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:10.466718 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:10.466803 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:10.466981 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:51:10.467246 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":174,"threads_started":1}
I20260928 02:51:10.468199 25407 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b3748d5ab641558812b6840f26159b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40251 } }
I20260928 02:51:10.468401 25407 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:10.468477 25407 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:10.468570 25407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:10.469623 25407 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b3748d5ab641558812b6840f26159b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40251 } }
I20260928 02:51:10.469969 25407 leader_election.cc:302] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8b3748d5ab641558812b6840f26159b; no voters:
I20260928 02:51:10.470366 25407 leader_election.cc:288] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:10.470392 25412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:10.470672 25412 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [term 1 LEADER]: Becoming Leader. State: Replica: c8b3748d5ab641558812b6840f26159b, State: Running, Role: LEADER
I20260928 02:51:10.471009 25412 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b3748d5ab641558812b6840f26159b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40251 } }
I20260928 02:51:10.471189 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:51:10.471743 25407 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:10.472849 25413 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8b3748d5ab641558812b6840f26159b. Latest consensus state: current_term: 1 leader_uuid: "c8b3748d5ab641558812b6840f26159b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b3748d5ab641558812b6840f26159b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40251 } } }
I20260928 02:51:10.472849 25414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8b3748d5ab641558812b6840f26159b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b3748d5ab641558812b6840f26159b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40251 } } }
I20260928 02:51:10.472990 25413 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:10.472990 25414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:10.473426 25421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:10.476344 25421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:10.482113 25421 catalog_manager.cc:1347] Generated new cluster ID: 80f51ed8f89343ad84285cc3e8eeaba3
I20260928 02:51:10.482223 25421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:10.494750 25421 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:10.495708 25421 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:10.501597 25421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8b3748d5ab641558812b6840f26159b: Generated new TSK 0
I20260928 02:51:10.502338 25421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:10.520813 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:10.515663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.19.139.190:40251)
0928 02:51:10.515798 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:10.515803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:10.515931 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:51:10.516099 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:10.516102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:10.516183 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:10.516300 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:51:10.516526 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.517356 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.518183 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.518530 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.518627 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.518917 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:51:10.520433 (+ 1516us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:51:10.520668 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:51:10.520890 25410 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41146: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:51:10.522384 25436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:10.515552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40251 (local address 127.0.0.1:41146)
0928 02:51:10.515848 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:10.515861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:10.515959 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:10.516301 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:10.516303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:10.516310 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:51:10.516431 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:10.516437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.517465 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.517468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:10.518083 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:10.518090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.518655 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.518658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:10.518723 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.518746 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:51:10.520586 (+ 1840us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:51:10.522218 (+ 1632us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.139.190:40251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37103/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1458,"thread_start_us":123,"threads_started":1}
I20260928 02:51:10.522971 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25340
E20260928 02:51:10.531536 25437 webserver.cc:533] Webserver: sslize failed:
E20260928 02:51:10.531622 25437 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T02:51:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:10Z Disabled control of system clock
I20260928 02:51:10.569180 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:34771
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:34771
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/testchainca.cert with env {}
W20260928 02:51:10.687913 25441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:10.688243 25441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:10.688293 25441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:10.692305 25441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:10.692369 25441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:10.692389 25441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:10.692417 25441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:10.692437 25441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:51:10.692458 25441 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:51:10.692549 25441 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:51:10.698189 25441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:34771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:34771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:10.699362 25441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:10.700563 25441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:10.709488 25447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:10.709583 25441 server_base.cc:1034] running on GCE node
W20260928 02:51:10.709497 25446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:10.709507 25449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:10.710328 25441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:10.711483 25441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:10.712642 25441 hybrid_clock.cc:648] HybridClock initialized: now 1790563870712624 us; error 33 us; skew 500 ppm
I20260928 02:51:10.714535 25441 webserver.cc:460] Webserver started at http://127.19.139.190:38287/ using document root <none> and password file <none>
I20260928 02:51:10.715112 25441 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:10.715178 25441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:10.715363 25441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:10.717536 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "f6f895a8da6c455984312edfaa7ecef3"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "ca60a54f99a793a589ea559803b53744"
server_key_iv: "8baf5c8900d5c28efef0d034c5438ee4"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.718108 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "f6f895a8da6c455984312edfaa7ecef3"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "ca60a54f99a793a589ea559803b53744"
server_key_iv: "8baf5c8900d5c28efef0d034c5438ee4"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.721982 25441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:10.724879 25455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:10.726357 25441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:10.726529 25441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "f6f895a8da6c455984312edfaa7ecef3"
format_stamp: "Formatted at 2026-09-28 02:51:10 on dist-test-slave-5vc0"
server_key: "ca60a54f99a793a589ea559803b53744"
server_key_iv: "8baf5c8900d5c28efef0d034c5438ee4"
server_key_version: "encryptionkey@0"
I20260928 02:51:10.726651 25441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:10.742358 25441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:10.743119 25441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:10.743302 25441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:10.750412 25441 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:34771
I20260928 02:51:10.750427 25507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:34771 every 8 connection(s)
I20260928 02:51:10.751405 25441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:10.754598 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:10.754938 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25441
I20260928 02:51:10.755052 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:10.755321 20014 external_mini_cluster.cc:1428] Setting key e04a8f65b38db98fa3c07fb2299f1d6e
I20260928 02:51:10.760648 25508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3: Bootstrap starting.
I20260928 02:51:10.763095 25508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:10.763872 25508 log.cc:824] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:10.764744 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:10.756771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.19.139.190:34771)
0928 02:51:10.757473 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:10.757489 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:10.757526 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:51:10.758168 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:10.758184 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:10.758597 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:10.758915 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:10.759344 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.760344 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.761246 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.761751 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.761898 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.762765 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:10.762771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:10.762786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:10.763171 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:10.763242 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:10.763340 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:10.763429 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:10.763559 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:51:10.763827 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260928 02:51:10.766045 25508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3: No bootstrap required, opened a new log
I20260928 02:51:10.767655 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:51:10.770224 25508 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f895a8da6c455984312edfaa7ecef3" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34771 } }
I20260928 02:51:10.770478 25508 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:10.770535 25508 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f895a8da6c455984312edfaa7ecef3, State: Initialized, Role: FOLLOWER
I20260928 02:51:10.770961 25508 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f895a8da6c455984312edfaa7ecef3" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34771 } }
I20260928 02:51:10.771088 25508 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:10.771152 25508 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:10.771251 25508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:10.772274 25508 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f895a8da6c455984312edfaa7ecef3" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34771 } }
I20260928 02:51:10.772616 25508 leader_election.cc:302] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f895a8da6c455984312edfaa7ecef3; no voters:
I20260928 02:51:10.772958 25508 leader_election.cc:288] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:10.773104 25513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:10.773357 25513 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [term 1 LEADER]: Becoming Leader. State: Replica: f6f895a8da6c455984312edfaa7ecef3, State: Running, Role: LEADER
I20260928 02:51:10.773664 25513 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f895a8da6c455984312edfaa7ecef3" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34771 } }
I20260928 02:51:10.774215 25508 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:10.775467 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f895a8da6c455984312edfaa7ecef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f895a8da6c455984312edfaa7ecef3" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34771 } } }
I20260928 02:51:10.775605 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:10.776187 25522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:10.776640 25515 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f895a8da6c455984312edfaa7ecef3. Latest consensus state: current_term: 1 leader_uuid: "f6f895a8da6c455984312edfaa7ecef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f895a8da6c455984312edfaa7ecef3" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34771 } } }
I20260928 02:51:10.776994 25515 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:10.779030 25522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:10.784795 25522 catalog_manager.cc:1347] Generated new cluster ID: c6018b037d4c4f82b608890b09429143
I20260928 02:51:10.784869 25522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:10.795486 25522 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:10.796824 25522 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:10.802603 25522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f895a8da6c455984312edfaa7ecef3: Generated new TSK 0
I20260928 02:51:10.803323 25522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:10.945534 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:10.819959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.19.139.190:34771)
0928 02:51:10.820089 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:10.820092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:10.820226 (+ 134us) server_negotiation.cc:407] Connection header received
0928 02:51:10.820410 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:10.820413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:10.820467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:10.820566 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:51:10.820829 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.821391 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.822636 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.822876 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.822962 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.823302 (+ 340us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:51:10.945121 (+121819us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:51:10.945408 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40616: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 02:51:10.945621 25511 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40616: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:51:10.945700 25537 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:10.819760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:34771 (local address 127.0.0.1:40616)
0928 02:51:10.820095 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:10.820111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:10.820271 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:10.820586 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:10.820589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:10.820599 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:51:10.820724 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:10.820731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.821492 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.821496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:10.822494 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:10.822504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:10.823001 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:10.823004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:10.823101 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:10.823130 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:51:10.945359 (+122229us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:51:10.945535 (+ 176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.139.190:34771: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":124,"threads_started":1}
I20260928 02:51:10.946290 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25441
2026-09-28T02:51:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (402 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:11 dist-test-slave-5vc0 krb5kdc[25549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:11 dist-test-slave-5vc0 krb5kdc[25549](info): set up 2 sockets
Sep 28 02:51:11 dist-test-slave-5vc0 krb5kdc[25549](info): commencing operation
krb5kdc: starting...
W20260928 02:51:12.144335 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:12.265525 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41351
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:12.376080 25562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:12.376446 25562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:12.376488 25562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:12.380473 25562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:12.380540 25562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:12.380559 25562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:12.380578 25562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:12.380592 25562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:12.386006 25562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:12.387056 25562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:12.388031 25562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:12.396219 25568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:12.396239 25567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:12.396415 25562 server_base.cc:1034] running on GCE node
W20260928 02:51:12.396219 25570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:12.397066 25562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:12.398207 25562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:12.399415 25562 hybrid_clock.cc:648] HybridClock initialized: now 1790563872399395 us; error 32 us; skew 500 ppm
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:12.405174 25562 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:12.406132 25562 webserver.cc:460] Webserver started at http://127.19.139.190:42257/ using document root <none> and password file <none>
I20260928 02:51:12.406679 25562 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:12.406744 25562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:12.406955 25562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:12.409060 25562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "0af659ff229c4094b08adc682a9819ba"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "4f934acc00cbd9150ddd991690740694"
server_key_iv: "8ea3858375265150b77560643acc6502"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.409605 25562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "0af659ff229c4094b08adc682a9819ba"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "4f934acc00cbd9150ddd991690740694"
server_key_iv: "8ea3858375265150b77560643acc6502"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.413273 25562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:51:12.415714 25577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.416989 25562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:51:12.417112 25562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "0af659ff229c4094b08adc682a9819ba"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "4f934acc00cbd9150ddd991690740694"
server_key_iv: "8ea3858375265150b77560643acc6502"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.417207 25562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:12.441372 25562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:12.445489 25562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:12.445686 25562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:12.453665 25562 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:41351
I20260928 02:51:12.453686 25629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:41351 every 8 connection(s)
I20260928 02:51:12.454914 25562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:12.458849 25630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:12.462126 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25562
I20260928 02:51:12.462262 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:12.462574 20014 external_mini_cluster.cc:1428] Setting key 65b960e62ae1f33f27f7b33cba5e2cbe
I20260928 02:51:12.468010 25630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba: Bootstrap starting.
I20260928 02:51:12.471261 25630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:12.472258 25630 log.cc:824] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:12.474798 25630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba: No bootstrap required, opened a new log
I20260928 02:51:12.479748 25630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af659ff229c4094b08adc682a9819ba" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41351 } }
I20260928 02:51:12.480095 25630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:12.480217 25630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af659ff229c4094b08adc682a9819ba, State: Initialized, Role: FOLLOWER
I20260928 02:51:12.480540 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:12.464465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.19.139.190:41351)
0928 02:51:12.464892 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:12.464912 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:12.464958 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:51:12.465703 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:12.465729 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:12.466216 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:12.466627 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:12.467769 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.469374 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.470282 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.470955 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.471115 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:12.474855 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:12.474862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:12.474881 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:12.474916 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:12.476915 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:12.477378 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:12.477385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:12.477391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:12.477530 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:12.477798 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:12.477802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:12.477804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:12.478319 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:12.478519 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:12.479120 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 02:51:12.479437 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260928 02:51:12.480767 25630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af659ff229c4094b08adc682a9819ba" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41351 } }
I20260928 02:51:12.480922 25630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:12.480988 25630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:12.481079 25630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:12.482291 25630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af659ff229c4094b08adc682a9819ba" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41351 } }
I20260928 02:51:12.482699 25630 leader_election.cc:302] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af659ff229c4094b08adc682a9819ba; no voters:
I20260928 02:51:12.483052 25630 leader_election.cc:288] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:12.483129 25635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:12.483294 25635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [term 1 LEADER]: Becoming Leader. State: Replica: 0af659ff229c4094b08adc682a9819ba, State: Running, Role: LEADER
I20260928 02:51:12.483588 25635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af659ff229c4094b08adc682a9819ba" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41351 } }
I20260928 02:51:12.484395 25630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:12.485169 25637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af659ff229c4094b08adc682a9819ba. Latest consensus state: current_term: 1 leader_uuid: "0af659ff229c4094b08adc682a9819ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af659ff229c4094b08adc682a9819ba" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41351 } } }
I20260928 02:51:12.485296 25637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:12.485736 25636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af659ff229c4094b08adc682a9819ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af659ff229c4094b08adc682a9819ba" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41351 } } }
I20260928 02:51:12.485864 25636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:12.486845 25643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:12.489619 25643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:12.495646 25643 catalog_manager.cc:1347] Generated new cluster ID: b0d38f13cf684302a1ed129ca6567ea4
I20260928 02:51:12.495723 25643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:12.517048 25643 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:12.518031 25643 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:12.529548 25643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af659ff229c4094b08adc682a9819ba: Generated new TSK 0
I20260928 02:51:12.530284 25643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:12.560616 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41351
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:12.677241 25656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:12.677603 25656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:12.677654 25656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:12.681990 25656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:12.682075 25656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:12.682142 25656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:12.687834 25656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:12.689005 25656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:12.690063 25656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:12.700030 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:12.700094 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:12.700065 25656 server_base.cc:1034] running on GCE node
W20260928 02:51:12.700341 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:12.700731 25656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:12.701406 25656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:12.702562 25656 hybrid_clock.cc:648] HybridClock initialized: now 1790563872702534 us; error 43 us; skew 500 ppm
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:12.705422 25656 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:12.706379 25656 webserver.cc:460] Webserver started at http://127.19.139.129:38849/ using document root <none> and password file <none>
I20260928 02:51:12.706915 25656 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:12.706976 25656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:12.707141 25656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:12.709196 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "3066f49eab894050843b8980c203b8e2"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "927264e344fc0b9b850682034987c217"
server_key_iv: "44e09579a7ba16b9c44921b3d62550f3"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.709714 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "3066f49eab894050843b8980c203b8e2"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "927264e344fc0b9b850682034987c217"
server_key_iv: "44e09579a7ba16b9c44921b3d62550f3"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.713656 25656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:12.716295 25671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.717721 25656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:12.717847 25656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "3066f49eab894050843b8980c203b8e2"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "927264e344fc0b9b850682034987c217"
server_key_iv: "44e09579a7ba16b9c44921b3d62550f3"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.717973 25656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:12.735623 25656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:12.738941 25656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:12.739130 25656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:12.739737 25656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:12.740617 25656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:12.740664 25656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.740705 25656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:12.740733 25656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.749279 25656 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:35219
I20260928 02:51:12.749312 25784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:35219 every 8 connection(s)
I20260928 02:51:12.750211 25656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:12.753285 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25656
I20260928 02:51:12.753413 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:12.753683 20014 external_mini_cluster.cc:1428] Setting key b8584ec96ed621b1af2ca82963ade83d
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:12.764132 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:12.752059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:50589 (local address 127.19.139.190:41351)
0928 02:51:12.752229 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:12.752233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:12.753116 (+ 883us) server_negotiation.cc:407] Connection header received
0928 02:51:12.754689 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:12.754693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:12.754745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:12.754826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:12.756039 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.756601 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.757736 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.758053 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.758152 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:12.760786 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:12.760789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:12.760791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:12.760815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:12.762106 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:12.762668 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:12.762671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:12.762672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:12.762745 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:12.763102 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:12.763106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:12.763107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:12.763295 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:12.763391 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:12.763892 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:51:12.764009 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:51:12.764997 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:12.752501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41351 (local address 127.19.139.129:50589)
0928 02:51:12.752956 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:12.753000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:12.754486 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:12.755018 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:12.755026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:12.755402 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:12.755886 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:12.755900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.756719 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.756721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:12.757614 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:12.757624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.758203 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.758206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:12.758377 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:12.758948 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:12.758971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:12.760590 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:12.762222 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:12.762229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:12.762247 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:12.762576 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:12.762854 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:12.762857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:12.762860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:12.762995 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:12.763403 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:12.763408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:12.763665 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 02:51:12.763846 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:51:12.764114 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:51:12.766216 25785 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41351
I20260928 02:51:12.766489 25785 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:12.767014 25785 heartbeater.cc:511] Master 127.19.139.190:41351 requested a full tablet report, sending...
I20260928 02:51:12.768563 25594 ts_manager.cc:194] Registered new tserver with Master: 3066f49eab894050843b8980c203b8e2 (127.19.139.129:35219)
I20260928 02:51:12.769694 25594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:50589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:12.780864 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41351
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:12.890909 25790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:12.891182 25790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:12.891239 25790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:12.895301 25790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:12.895390 25790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:12.895493 25790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:12.900955 25790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:12.902050 25790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:12.903084 25790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:12.912647 25798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:12.912768 25790 server_base.cc:1034] running on GCE node
W20260928 02:51:12.912632 25796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:12.912632 25795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:12.913335 25790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:12.913964 25790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:12.915118 25790 hybrid_clock.cc:648] HybridClock initialized: now 1790563872915102 us; error 29 us; skew 500 ppm
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:12.917773 25790 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:12.918602 25790 webserver.cc:460] Webserver started at http://127.19.139.130:39929/ using document root <none> and password file <none>
I20260928 02:51:12.919112 25790 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:12.919167 25790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:12.919341 25790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:12.921490 25790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "e570fb29925a41d6b6b0e38eb5fd4171"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "8b4b7f086da85167fa1cf934d6266db9"
server_key_iv: "75a4eed99c5afdb54d2c0f0e9a6ad9b4"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.922068 25790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "e570fb29925a41d6b6b0e38eb5fd4171"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "8b4b7f086da85167fa1cf934d6266db9"
server_key_iv: "75a4eed99c5afdb54d2c0f0e9a6ad9b4"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.925762 25790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:51:12.928222 25805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.929452 25790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:12.929581 25790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "e570fb29925a41d6b6b0e38eb5fd4171"
format_stamp: "Formatted at 2026-09-28 02:51:12 on dist-test-slave-5vc0"
server_key: "8b4b7f086da85167fa1cf934d6266db9"
server_key_iv: "75a4eed99c5afdb54d2c0f0e9a6ad9b4"
server_key_version: "encryptionkey@0"
I20260928 02:51:12.929684 25790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:12.958827 25790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:12.962123 25790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:12.962304 25790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:12.962909 25790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:12.963733 25790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:12.963778 25790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.963819 25790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:12.963846 25790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:12.972214 25790 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:36009
I20260928 02:51:12.972257 25918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:36009 every 8 connection(s)
I20260928 02:51:12.973140 25790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:12.977267 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25790
I20260928 02:51:12.977396 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:12.977705 20014 external_mini_cluster.cc:1428] Setting key a161552247827b4dd036d31efc0c4793
Sep 28 02:51:12 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563872, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:51:12.988111 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:12.975022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:54705 (local address 127.19.139.190:41351)
0928 02:51:12.975182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:12.975185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:12.976077 (+ 892us) server_negotiation.cc:407] Connection header received
0928 02:51:12.977551 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:12.977555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:12.977618 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:12.977711 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:12.979224 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.979753 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.981312 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.981557 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.981658 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:12.984335 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:12.984340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:12.984343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:12.984395 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:12.985988 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:12.986571 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:12.986575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:12.986577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:12.986671 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:12.987014 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:12.987017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:12.987019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:12.987277 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:12.987378 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:12.987860 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 02:51:12.987985 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:51:12.989018 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:12.975383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41351 (local address 127.19.139.130:54705)
0928 02:51:12.975890 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:12.975935 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:12.977307 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:12.977915 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:12.977923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:12.978443 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:12.979027 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:12.979048 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.979893 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.979896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:12.981161 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:12.981175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:12.981718 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:12.981721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:12.981893 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:12.982490 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:12.982508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:12.984114 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:12.986115 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:12.986123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:12.986136 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:12.986465 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:12.986773 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:12.986775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:12.986777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:12.986918 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:12.987371 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:12.987376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:12.987631 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 02:51:12.987800 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:51:12.988085 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:51:12.990290 25919 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41351
I20260928 02:51:12.990556 25919 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:12.991140 25919 heartbeater.cc:511] Master 127.19.139.190:41351 requested a full tablet report, sending...
I20260928 02:51:12.992274 25594 ts_manager.cc:194] Registered new tserver with Master: e570fb29925a41d6b6b0e38eb5fd4171 (127.19.139.130:36009)
I20260928 02:51:12.992839 25594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:54705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:13.007618 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41351
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:13.121600 25924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:13.121877 25924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:13.121937 25924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:13.126056 25924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:13.126123 25924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:13.126205 25924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:13.131595 25924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:13.132817 25924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:13.133912 25924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:13.143867 25930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:13.143898 25932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:13.144194 25924 server_base.cc:1034] running on GCE node
W20260928 02:51:13.144289 25929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:13.144724 25924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:13.145452 25924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:13.146631 25924 hybrid_clock.cc:648] HybridClock initialized: now 1790563873146608 us; error 33 us; skew 500 ppm
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.149402 25924 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:13.150363 25924 webserver.cc:460] Webserver started at http://127.19.139.131:38659/ using document root <none> and password file <none>
I20260928 02:51:13.150914 25924 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:13.150985 25924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:13.151196 25924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:13.153357 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "28111bcf9f0840fd944a44a1a0e9b650"
format_stamp: "Formatted at 2026-09-28 02:51:13 on dist-test-slave-5vc0"
server_key: "1eaa29f173ce3be5f4f561d920fb8f18"
server_key_iv: "22f4bd8e7634f4123ffb862f4de7df06"
server_key_version: "encryptionkey@0"
I20260928 02:51:13.153908 25924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "28111bcf9f0840fd944a44a1a0e9b650"
format_stamp: "Formatted at 2026-09-28 02:51:13 on dist-test-slave-5vc0"
server_key: "1eaa29f173ce3be5f4f561d920fb8f18"
server_key_iv: "22f4bd8e7634f4123ffb862f4de7df06"
server_key_version: "encryptionkey@0"
I20260928 02:51:13.157982 25924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:51:13.160497 25939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:13.161710 25924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:51:13.161813 25924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "28111bcf9f0840fd944a44a1a0e9b650"
format_stamp: "Formatted at 2026-09-28 02:51:13 on dist-test-slave-5vc0"
server_key: "1eaa29f173ce3be5f4f561d920fb8f18"
server_key_iv: "22f4bd8e7634f4123ffb862f4de7df06"
server_key_version: "encryptionkey@0"
I20260928 02:51:13.161916 25924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:13.179191 25924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:13.182590 25924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:13.182771 25924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:13.183393 25924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:13.184291 25924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:13.184337 25924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:13.184406 25924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:13.184437 25924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:13.194453 25924 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:44707
I20260928 02:51:13.194485 26052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:44707 every 8 connection(s)
I20260928 02:51:13.195390 25924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:13.195741 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 25924
I20260928 02:51:13.195873 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:13.196309 20014 external_mini_cluster.cc:1428] Setting key 348003db59e411cfdedf4bf30ad1a532
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:13.209499 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:13.197412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:38899 (local address 127.19.139.190:41351)
0928 02:51:13.197545 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.197549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.199211 (+ 1662us) server_negotiation.cc:407] Connection header received
0928 02:51:13.200596 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.200600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.200655 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.200726 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.201911 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.202433 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.203570 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.203793 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.203877 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:13.206234 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:13.206238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:13.206241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:13.206269 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:13.207438 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.207943 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.207946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.207948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.208023 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.208377 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.208380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.208382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.208568 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:13.208650 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:13.209208 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 02:51:13.209330 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:51:13.210417 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:13.198010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41351 (local address 127.19.139.131:38899)
0928 02:51:13.199082 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:13.199124 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:13.200389 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:13.200903 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:13.200911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:13.201276 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:13.201777 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:13.201790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.202563 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.202565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:13.203455 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:13.203465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.203937 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.203941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:13.204109 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:13.204686 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:13.204702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:13.206076 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:13.207583 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:13.207589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:13.207606 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:13.207853 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:13.208103 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:13.208106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:13.208108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:13.208292 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:13.208661 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:13.208668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:13.208953 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 02:51:13.209150 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:51:13.209476 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":138,"threads_started":1}
I20260928 02:51:13.211476 26053 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41351
I20260928 02:51:13.211699 26053 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:13.212216 26053 heartbeater.cc:511] Master 127.19.139.190:41351 requested a full tablet report, sending...
I20260928 02:51:13.213187 25594 ts_manager.cc:194] Registered new tserver with Master: 28111bcf9f0840fd944a44a1a0e9b650 (127.19.139.131:44707)
I20260928 02:51:13.213711 25594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:38899
I20260928 02:51:13.222649 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/test-admin.keytab.
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.236815 20014 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:51:13.239522 26060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:13.238128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.19.139.129:35219)
0928 02:51:13.238435 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.238450 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.238496 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:51:13.238548 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.238553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.238741 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.238861 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.239321 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42662: BlockingRecv error: recv got EOF from 127.0.0.1:42662 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
W20260928 02:51:13.239598 20014 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.240625 20014 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.241518 20014 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:51:13.250977 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:13.242456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.19.139.129:35219)
0928 02:51:13.242571 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.242576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.242639 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:51:13.242770 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.242773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.242836 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.242898 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.243455 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.244313 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.245123 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.245520 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.245613 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:13.247763 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:13.247791 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:13.247805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:13.247829 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:13.249244 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.249674 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.249681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.249682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.249763 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.250019 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.250022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.250024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.250268 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:13.250458 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:13.250671 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 02:51:13.250831 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:51:13.254701 26060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:13.253837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.19.139.129:35219)
0928 02:51:13.253962 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.253966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.254031 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:51:13.254115 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.254117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.254153 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.254214 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.254593 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42686: BlockingRecv error: recv got EOF from 127.0.0.1:42686 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 02:51:13.254854 20014 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.255806 20014 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.256673 20014 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:51:13.265146 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:13.257534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.19.139.129:35219)
0928 02:51:13.257666 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.257669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.257685 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:51:13.257800 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.257803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.257838 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.257912 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.258438 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.258962 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.259794 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.260027 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.260102 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:13.262291 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:13.262294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:13.262296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:13.262318 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:13.263662 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.264064 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.264067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.264068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.264141 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.264402 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.264406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.264407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.264566 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:13.264675 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:13.264890 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 02:51:13.264994 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 02:51:13.267774 26060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:13.266836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.19.139.129:35219)
0928 02:51:13.266956 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.266960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.267047 (+ 87us) server_negotiation.cc:407] Connection header received
0928 02:51:13.267106 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.267108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.267143 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.267203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.267679 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42710: BlockingRecv error: recv got EOF from 127.0.0.1:42710 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 02:51:13.267820 20014 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563845681472-20014-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.268806 20014 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:13.269642 20014 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[25549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:51:13.277864 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:13.270405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.19.139.129:35219)
0928 02:51:13.270522 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:13.270525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:13.270588 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:51:13.270668 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:13.270671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:13.270702 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:13.270774 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:13.271179 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.271683 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.272538 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:13.272765 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:13.272841 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:13.274954 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:13.274957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:13.274959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:13.274981 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:13.276399 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.276780 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.276784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.276785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.276854 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:13.277090 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:13.277093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:13.277095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:13.277255 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:13.277364 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:13.277609 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:51:13.277705 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:51:13.278787 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25656
I20260928 02:51:13.283740 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25790
I20260928 02:51:13.288316 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25924
I20260928 02:51:13.292474 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 25562
2026-09-28T02:51:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2100 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[26073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[26073](info): set up 2 sockets
Sep 28 02:51:13 dist-test-slave-5vc0 krb5kdc[26073](info): commencing operation
krb5kdc: starting...
W20260928 02:51:14.252281 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:14 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab.
Entry for principal oryx/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab.
I20260928 02:51:14.372196 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:33543
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab
--principal=oryx/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:14.483857 26086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:14.484200 26086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:14.484280 26086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:14.488306 26086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:14.488373 26086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:14.488394 26086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:14.488413 26086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:14.488430 26086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:14.493886 26086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:33543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab
--principal=oryx/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:14.494974 26086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:14.496031 26086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:14.504705 26091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:14.504719 26092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:14.504757 26086 server_base.cc:1034] running on GCE node
W20260928 02:51:14.504729 26094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:14.505452 26086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:14.506575 26086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:14.507751 26086 hybrid_clock.cc:648] HybridClock initialized: now 1790563874507735 us; error 28 us; skew 500 ppm
Sep 28 02:51:14 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:14.514024 26086 init.cc:377] Logged in from keytab as oryx/127.19.139.190@KRBTEST.COM (short username oryx)
I20260928 02:51:14.515046 26086 webserver.cc:460] Webserver started at http://127.19.139.190:43523/ using document root <none> and password file <none>
I20260928 02:51:14.515614 26086 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:14.515681 26086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:14.515883 26086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:14.518088 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "7064d87b6446440eb11e70ad20437c70"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "ed34bbec0d925020e80d0e35a0823e58"
server_key_iv: "546175e567ca0892a582740d5967e383"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.518657 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "7064d87b6446440eb11e70ad20437c70"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "ed34bbec0d925020e80d0e35a0823e58"
server_key_iv: "546175e567ca0892a582740d5967e383"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.522684 26086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:14.525372 26101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:14.526757 26086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:51:14.526922 26086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "7064d87b6446440eb11e70ad20437c70"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "ed34bbec0d925020e80d0e35a0823e58"
server_key_iv: "546175e567ca0892a582740d5967e383"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.527042 26086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:14.539598 26086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:14.542575 26086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:14.542762 26086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:14.549703 26086 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:33543
I20260928 02:51:14.549957 26153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:33543 every 8 connection(s)
I20260928 02:51:14.550724 26086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:14.553537 26154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:14.558718 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26086
I20260928 02:51:14.558835 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:14.559106 20014 external_mini_cluster.cc:1428] Setting key c71e91c627b87a0ac227241f8aa81472
I20260928 02:51:14.559309 26154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Bootstrap starting.
I20260928 02:51:14.561868 26154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:14.562583 26154 log.cc:824] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:14.564500 26154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: No bootstrap required, opened a new log
Sep 28 02:51:14 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:14.568296 26154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } }
I20260928 02:51:14.568595 26154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:14.568660 26154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7064d87b6446440eb11e70ad20437c70, State: Initialized, Role: FOLLOWER
I20260928 02:51:14.569137 26154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } }
I20260928 02:51:14.569288 26154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:14.569357 26154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:14.569451 26154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:14.570359 26154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } }
I20260928 02:51:14.570726 26154 leader_election.cc:302] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7064d87b6446440eb11e70ad20437c70; no voters:
I20260928 02:51:14.571046 26154 leader_election.cc:288] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:14.571221 26159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:14.571463 26159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 LEADER]: Becoming Leader. State: Replica: 7064d87b6446440eb11e70ad20437c70, State: Running, Role: LEADER
I20260928 02:51:14.571780 26159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } }
I20260928 02:51:14.572429 26154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:14.572973 26159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7064d87b6446440eb11e70ad20437c70. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } }
I20260928 02:51:14.573084 26159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:14.573367 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } }
I20260928 02:51:14.573452 26164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:14.573472 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:14.574198 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:14.560504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.19.139.190:33543)
0928 02:51:14.560981 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:14.561000 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:14.561037 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:51:14.561701 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:14.561720 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:14.562109 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:14.562432 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:14.563222 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:14.564256 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:14.565132 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:14.565608 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:14.565755 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:14.568612 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:14.568618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:14.568636 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:14.568670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:14.570383 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:14.570889 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:14.570896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:14.570902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:14.571024 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:14.571347 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:14.571351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:14.571352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:14.571854 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:14.572052 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:14.572667 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 02:51:14.572993 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260928 02:51:14.576710 26164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:14.582449 26164 catalog_manager.cc:1347] Generated new cluster ID: 6db09412b7294458b363ab483943b76a
I20260928 02:51:14.582541 26164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:14.588284 26164 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:14.589586 26164 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:14.597733 26164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Generated new TSK 0
I20260928 02:51:14.598734 26164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab.
Entry for principal oryx/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab.
I20260928 02:51:14.631470 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33543
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:14.743954 26180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:14.744313 26180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:14.744364 26180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:14.748420 26180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:14.748483 26180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:14.748543 26180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:14.753851 26180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:14.754956 26180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:14.756033 26180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:14.765731 26185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:14.765982 26180 server_base.cc:1034] running on GCE node
W20260928 02:51:14.765731 26186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:14.765741 26188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:14.766485 26180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:14.767138 26180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:14.768280 26180 hybrid_clock.cc:648] HybridClock initialized: now 1790563874768266 us; error 31 us; skew 500 ppm
Sep 28 02:51:14 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:14.770905 26180 init.cc:377] Logged in from keytab as oryx/127.19.139.129@KRBTEST.COM (short username oryx)
I20260928 02:51:14.771730 26180 webserver.cc:460] Webserver started at http://127.19.139.129:39851/ using document root <none> and password file <none>
I20260928 02:51:14.772291 26180 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:14.772362 26180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:14.772557 26180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:14.774634 26180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "9224ae22ca9c41dcbabe9026ec7699aa"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "6a54ca76da86f2f43f95ae5e3a3bea14"
server_key_iv: "3e77871b55bd64cac4b25e25de66c570"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.775182 26180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "9224ae22ca9c41dcbabe9026ec7699aa"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "6a54ca76da86f2f43f95ae5e3a3bea14"
server_key_iv: "3e77871b55bd64cac4b25e25de66c570"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.778949 26180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:51:14.781518 26195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:14.782824 26180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:14.782959 26180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "9224ae22ca9c41dcbabe9026ec7699aa"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "6a54ca76da86f2f43f95ae5e3a3bea14"
server_key_iv: "3e77871b55bd64cac4b25e25de66c570"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.783070 26180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:14.796572 26180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:14.799734 26180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:14.799924 26180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:14.800652 26180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:14.801542 26180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:14.801592 26180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:14.801644 26180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:14.801669 26180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:14.810137 26180 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:45839
I20260928 02:51:14.810165 26308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:45839 every 8 connection(s)
I20260928 02:51:14.811089 26180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:14.818274 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26180
I20260928 02:51:14.818435 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:14.818709 20014 external_mini_cluster.cc:1428] Setting key 407ee05cf0acd8de15bf84741011c03e
Sep 28 02:51:14 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:14.825742 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:14.812885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:50675 (local address 127.19.139.190:33543)
0928 02:51:14.813033 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:14.813038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:14.813963 (+ 925us) server_negotiation.cc:407] Connection header received
0928 02:51:14.815324 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:14.815328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:14.815474 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:14.815571 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:14.817144 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:14.817677 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:14.818895 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:14.819200 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:14.819304 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:14.822014 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:14.822018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:14.822021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:14.822051 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:14.823511 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:14.824033 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:14.824037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:14.824038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:14.824138 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:14.824525 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:14.824529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:14.824530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:14.824771 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:14.824879 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:14.825424 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:51:14.825559 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:51:14.826738 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:14.813327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:50675)
0928 02:51:14.813791 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:14.813837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:14.815099 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:14.815863 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:14.815874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:14.816465 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:14.816997 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:14.817012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:14.817789 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:14.817791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:14.818768 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:14.818780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:14.819352 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:14.819357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:14.819584 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:14.820260 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:14.820278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:14.821822 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:14.823633 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:14.823638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:14.823653 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:14.823927 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:14.824249 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:14.824252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:14.824253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:14.824436 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:14.824904 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:14.824910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:14.825172 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 02:51:14.825354 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 02:51:14.825645 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:14.828042 26309 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33543
I20260928 02:51:14.828346 26309 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:14.828841 26309 heartbeater.cc:511] Master 127.19.139.190:33543 requested a full tablet report, sending...
I20260928 02:51:14.830423 26118 ts_manager.cc:194] Registered new tserver with Master: 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.130@KRBTEST.COM" created.
I20260928 02:51:14.831645 26118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:50675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab.
Entry for principal oryx/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab.
I20260928 02:51:14.846500 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33543
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:14.955507 26314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:14.955792 26314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:14.955852 26314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:14.960004 26314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:14.960076 26314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:14.960175 26314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:14.965646 26314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:14.966729 26314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:14.967787 26314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:14.977347 26319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:14.977349 26320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:14.977349 26322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:14.977618 26314 server_base.cc:1034] running on GCE node
I20260928 02:51:14.978116 26314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:14.978786 26314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:14.979990 26314 hybrid_clock.cc:648] HybridClock initialized: now 1790563874979977 us; error 47 us; skew 500 ppm
Sep 28 02:51:14 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:14.983024 26314 init.cc:377] Logged in from keytab as oryx/127.19.139.130@KRBTEST.COM (short username oryx)
I20260928 02:51:14.984011 26314 webserver.cc:460] Webserver started at http://127.19.139.130:42185/ using document root <none> and password file <none>
I20260928 02:51:14.984635 26314 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:14.984710 26314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:14.984933 26314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:14.987110 26314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "bc8630532d7243439983b8cff12fa436"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "735ea0cb2f85e8f56f7ccb071cc05cea"
server_key_iv: "de7c4283606eed6e7368d405f4c18626"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.987663 26314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "bc8630532d7243439983b8cff12fa436"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "735ea0cb2f85e8f56f7ccb071cc05cea"
server_key_iv: "de7c4283606eed6e7368d405f4c18626"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.991671 26314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 02:51:14.994210 26329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:14.995453 26314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:14.995572 26314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "bc8630532d7243439983b8cff12fa436"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "735ea0cb2f85e8f56f7ccb071cc05cea"
server_key_iv: "de7c4283606eed6e7368d405f4c18626"
server_key_version: "encryptionkey@0"
I20260928 02:51:14.995672 26314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.016755 26314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:15.020224 26314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:15.020434 26314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:15.021072 26314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:15.021961 26314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:15.022010 26314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.022064 26314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:15.022092 26314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.030942 26314 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:41777
I20260928 02:51:15.030987 26442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:41777 every 8 connection(s)
I20260928 02:51:15.031926 26314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:15.032701 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26314
I20260928 02:51:15.032807 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:15.033134 20014 external_mini_cluster.cc:1428] Setting key 59748ae105afc2df4556e12d36ea76c0
WARNING: no policy specified for oryx/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.131@KRBTEST.COM" created.
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:15.050686 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.033799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:49459 (local address 127.19.139.190:33543)
0928 02:51:15.033930 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.033934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.036483 (+ 2549us) server_negotiation.cc:407] Connection header received
0928 02:51:15.037869 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.037873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.037929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.038012 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.039312 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.040140 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.041319 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.041673 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.041788 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.046829 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.046833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.046835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.046867 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.048286 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.048925 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.048928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.048929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.049009 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.049442 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.049445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.049447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.049652 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.049732 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.050442 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.050554 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:51:15.051993 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.034349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:49459)
0928 02:51:15.036326 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.036376 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.037647 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.038206 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.038214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.038580 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.039142 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.039162 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.040280 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.040282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.041188 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.041197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.041819 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.041822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.041968 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.042588 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.042604 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.046592 (+ 3988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.048423 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.048430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.048451 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.048829 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.049090 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.049093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.049095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.049318 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.049760 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.049767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.050169 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.050425 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.050835 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1740,"thread_start_us":129,"threads_started":1}
I20260928 02:51:15.053215 26443 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33543
I20260928 02:51:15.053467 26443 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:15.053990 26443 heartbeater.cc:511] Master 127.19.139.190:33543 requested a full tablet report, sending...
I20260928 02:51:15.055100 26118 ts_manager.cc:194] Registered new tserver with Master: bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777)
I20260928 02:51:15.055702 26118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:49459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab.
Entry for principal oryx/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab.
I20260928 02:51:15.063283 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33543
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:15.174621 26448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:15.174921 26448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:15.174983 26448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:15.179327 26448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:15.179409 26448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:15.179492 26448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:15.185132 26448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:15.186252 26448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:15.187311 26448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:15.197141 26456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:15.197218 26453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:15.197141 26454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.197304 26448 server_base.cc:1034] running on GCE node
I20260928 02:51:15.197837 26448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:15.198498 26448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:15.199654 26448 hybrid_clock.cc:648] HybridClock initialized: now 1790563875199622 us; error 54 us; skew 500 ppm
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:15.202498 26448 init.cc:377] Logged in from keytab as oryx/127.19.139.131@KRBTEST.COM (short username oryx)
I20260928 02:51:15.203366 26448 webserver.cc:460] Webserver started at http://127.19.139.131:37863/ using document root <none> and password file <none>
I20260928 02:51:15.203888 26448 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.203946 26448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.204120 26448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:15.206192 26448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "38f75d9c849745b999b55d607315f5a3"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "af608b1b2bbd2b3b2ef194432fc97bda"
server_key_iv: "8d59ede06dc38b05c1a57ab2b773f7c9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.206743 26448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "38f75d9c849745b999b55d607315f5a3"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "af608b1b2bbd2b3b2ef194432fc97bda"
server_key_iv: "8d59ede06dc38b05c1a57ab2b773f7c9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.210709 26448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:15.213205 26463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.214475 26448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:15.214581 26448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "38f75d9c849745b999b55d607315f5a3"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "af608b1b2bbd2b3b2ef194432fc97bda"
server_key_iv: "8d59ede06dc38b05c1a57ab2b773f7c9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.214701 26448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.239414 26448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:15.242599 26448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:15.242774 26448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:15.243361 26448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:15.244218 26448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:15.244263 26448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.244305 26448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:15.244333 26448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.252660 26448 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46441
I20260928 02:51:15.252696 26576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46441 every 8 connection(s)
I20260928 02:51:15.253549 26448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:15.259930 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26448
I20260928 02:51:15.260044 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:15.260332 20014 external_mini_cluster.cc:1428] Setting key 854aa1310197011104dbbe6905e351f0
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:15.267024 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.255367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:51391 (local address 127.19.139.190:33543)
0928 02:51:15.255526 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.255531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.256444 (+ 913us) server_negotiation.cc:407] Connection header received
0928 02:51:15.257738 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.257741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.257787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.257860 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.259235 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.259784 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.261146 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.261426 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.261544 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.264011 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.264014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.264016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.264042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.265204 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.265704 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.265706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.265708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.265781 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.266053 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.266055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.266056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.266225 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.266295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.266804 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.266899 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:51:15.267915 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.255769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:51391)
0928 02:51:15.256264 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.256345 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.257553 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.258056 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.258063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.258495 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.259020 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.259057 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.259915 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.259918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.261026 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.261036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.261573 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.261578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.261771 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.262430 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.262447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.263847 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.265303 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.265307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.265321 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.265627 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.265843 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.265845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.265847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.265979 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.266314 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.266319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.266612 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.266776 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.267048 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260928 02:51:15.269078 26577 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33543
I20260928 02:51:15.269392 26577 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:15.269901 26577 heartbeater.cc:511] Master 127.19.139.190:33543 requested a full tablet report, sending...
I20260928 02:51:15.270929 26118 ts_manager.cc:194] Registered new tserver with Master: 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441)
I20260928 02:51:15.271587 26118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:51391
I20260928 02:51:15.274147 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.139.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.189.keytab.
Entry for principal oryx/127.19.139.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.189.keytab.
I20260928 02:51:15.299706 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:33411
--webserver_interface=127.19.139.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.189.keytab
--principal=oryx/127.19.139.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:33543,127.19.139.189:33411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:15.411911 26582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:15.412211 26582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:15.412258 26582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:15.416358 26582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:15.416425 26582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:15.416445 26582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:15.416474 26582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:15.416502 26582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:15.422119 26582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:33543,127.19.139.189:33411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.189.keytab
--principal=oryx/127.19.139.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.139.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
W20260928 02:51:15.422389 26582 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.139.190:33543,127.19.139.189:33411'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:51:15.423329 26582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:15.424479 26582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:15.433218 26590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:15.433223 26587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.433538 26582 server_base.cc:1034] running on GCE node
W20260928 02:51:15.433223 26588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.434212 26582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:15.435364 26582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:15.436520 26582 hybrid_clock.cc:648] HybridClock initialized: now 1790563875436495 us; error 40 us; skew 500 ppm
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:15.439278 26582 init.cc:377] Logged in from keytab as oryx/127.19.139.189@KRBTEST.COM (short username oryx)
I20260928 02:51:15.440289 26582 webserver.cc:460] Webserver started at http://127.19.139.189:43575/ using document root <none> and password file <none>
I20260928 02:51:15.440838 26582 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.440897 26582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.441062 26582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:15.443116 26582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/instance:
uuid: "105043607d974614bb4289128ca69849"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "40f555c3f85cb6ea664426da1905f965"
server_key_iv: "d5315d841180918ca9e6d83b90ba8165"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.443640 26582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal/instance:
uuid: "105043607d974614bb4289128ca69849"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "40f555c3f85cb6ea664426da1905f965"
server_key_iv: "d5315d841180918ca9e6d83b90ba8165"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.447458 26582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:51:15.450086 26597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.451366 26582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:15.451478 26582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
uuid: "105043607d974614bb4289128ca69849"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "40f555c3f85cb6ea664426da1905f965"
server_key_iv: "d5315d841180918ca9e6d83b90ba8165"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.451592 26582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.471212 26582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:15.474686 26582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:15.474862 26582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:15.481571 26582 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.189:33411
I20260928 02:51:15.481814 26649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.189:33411 every 8 connection(s)
I20260928 02:51:15.482555 26582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
I20260928 02:51:15.485316 26650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:15.486480 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26582
I20260928 02:51:15.489650 26650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.189@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:15.500882 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.491617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.19.139.190:33543)
0928 02:51:15.491758 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.491763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.492077 (+ 314us) server_negotiation.cc:407] Connection header received
0928 02:51:15.492228 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.492231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.492282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.492342 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.492995 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.493786 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.494731 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.494943 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.495029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.497407 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.497413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.497416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.497448 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.498684 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.499313 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.499315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.499317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.499428 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.499727 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.499730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.499732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.499967 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.500053 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.500586 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.500700 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:51:15.501276 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.491532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:57522)
0928 02:51:15.491950 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.491997 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.492112 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.492408 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.492414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.492759 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.492881 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.492887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.493978 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.493981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.494613 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.494622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.495076 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.495080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.495176 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.495630 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.495652 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.497204 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.498968 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.498975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.498981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.499218 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.499561 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.499563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.499565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.499642 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.500070 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.500078 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.500373 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.500515 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.500703 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":146,"threads_started":1}
I20260928 02:51:15.501282 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.487930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.19.139.189:33411)
0928 02:51:15.488365 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.488395 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.488591 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:51:15.489477 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.489502 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.489938 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.490313 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.490961 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.492336 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.493257 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.493747 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.493892 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.496570 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.496575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.496585 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.496606 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.498150 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.498575 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.498579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.498585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.498689 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.498954 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.498958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.498960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.499531 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.499741 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.500029 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.500390 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20260928 02:51:15.503796 26650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.189@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
I20260928 02:51:15.511493 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.504303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56628)
0928 02:51:15.504477 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.504489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.504561 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.504749 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.504751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.504904 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.504997 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.505002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.506102 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.506104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.506673 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.506682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.507169 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.507172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.507261 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.507755 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.507766 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.509010 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.510342 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.510344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.510346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.510529 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.510725 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.510727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.510729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.510792 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.511082 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.511084 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.511154 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.511224 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.511324 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:51:15.511513 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.504374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.19.139.189:33411)
0928 02:51:15.504482 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.504485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.504525 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:51:15.504625 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.504627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.504661 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.504727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.505097 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.505981 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.506798 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.507043 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.507129 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.509133 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.509136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.509138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.509159 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.510249 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.510599 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.510601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.510602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.510666 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.510847 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.510849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.510850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.511022 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.511090 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.511264 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.511378 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260928 02:51:15.514410 26650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Bootstrap starting.
I20260928 02:51:15.516831 26650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:15.517580 26650 log.cc:824] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:15.519644 26650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: No bootstrap required, opened a new log
I20260928 02:51:15.522677 26650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } }
I20260928 02:51:15.522903 26650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:15.522943 26650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105043607d974614bb4289128ca69849, State: Initialized, Role: FOLLOWER
I20260928 02:51:15.523378 26650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } }
I20260928 02:51:15.523883 26656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } } }
I20260928 02:51:15.524006 26656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:15.524657 26650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:51:15.529281 26667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:51:15.529371 26667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:51:15.531235 26582 master_runner.cc:428] Detected that this master 105043607d974614bb4289128ca69849 is joining an existing cluster
I20260928 02:51:15.531297 26582 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.139.189:33411
I20260928 02:51:15.541461 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.533744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.19.139.190:33543)
0928 02:51:15.533875 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.533880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.534106 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:51:15.534158 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.534161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.534211 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.534298 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.534841 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.535498 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.536795 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.537129 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.537216 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.538426 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.538430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.538433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.538459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.539793 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540280 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.540284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.540286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.540384 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540650 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.540653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.540655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.540843 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.540931 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.541195 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.541315 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:51:15.541476 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.533669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:57536)
0928 02:51:15.533965 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.533978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.534075 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.534312 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.534315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.534563 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.534694 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.534701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.535608 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.535611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.536685 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.536695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.537365 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.537368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.537442 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.537900 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.537912 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.538288 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.539900 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.539907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.539909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.540125 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.540473 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.540478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.540567 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.540931 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.540934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.541045 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.541130 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.541276 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":43,"thread_start_us":79,"threads_started":1}
I20260928 02:51:15.541690 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.534450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.19.139.189:33411)
0928 02:51:15.534544 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.534548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.534558 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:51:15.534583 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.534585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.534618 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.534690 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.535207 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.535842 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.536720 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.537009 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.537104 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.538454 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.538458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.538460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.538481 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.540095 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540538 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.540541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.540543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.540633 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540878 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.540881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.540883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.541092 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.541158 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.541433 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.541539 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":4480}
I20260928 02:51:15.542146 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.534072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56642)
0928 02:51:15.534298 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.534310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.534385 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.534727 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.534730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.534954 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.535094 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.535102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.535951 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.535954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.536624 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.536633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.537123 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.537127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.537223 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.537777 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.537792 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.538327 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.540226 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.540232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.540431 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.540722 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.540724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.540726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.540800 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.541210 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.541212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.541292 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.541850 (+ 558us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.541990 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":33,"thread_start_us":72,"threads_started":1}
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.190@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
I20260928 02:51:15.556007 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.545944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56654)
0928 02:51:15.546284 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.546345 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.546481 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.546795 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.546800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.547158 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.547279 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.547285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.548267 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.548270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.548924 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.548932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.549550 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.549554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.549662 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.550249 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.550264 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.552046 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.554009 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.554019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.554026 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.554390 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.554674 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.554677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.554679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.554766 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.555209 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.555216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.555450 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.555601 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.555817 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260928 02:51:15.556680 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.545976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.19.139.189:33411)
0928 02:51:15.546083 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.546087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.546415 (+ 328us) server_negotiation.cc:407] Connection header received
0928 02:51:15.546627 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.546630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.546680 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.546752 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.547374 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.548150 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.549020 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.549351 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.549466 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.552240 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.552244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.552247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.552272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.553889 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.554495 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.554499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.554501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.554584 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.554858 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.554861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.554863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.555075 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.555159 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.556420 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.556538 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:51:15.558274 26118 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.139.189:33411: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "105043607d974614bb4289128ca69849"
member_type: NON_VOTER
last_known_addr {
host: "127.19.139.189"
port: 33411
}
attrs {
promote: true
}
}
dest_uuid: "7064d87b6446440eb11e70ad20437c70"
cas_config_opid_index: -1
I20260928 02:51:15.558985 26118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } }
I20260928 02:51:15.561713 26678 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 105043607d974614bb4289128ca69849 (127.19.139.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } }
I20260928 02:51:15.562000 26624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:15.563431 26624 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Refusing update from remote peer 7064d87b6446440eb11e70ad20437c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:51:15.563582 26678 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.139.189:33411
I20260928 02:51:15.564011 26582 master.cc:537] Master@127.19.139.189:33411 shutting down...
I20260928 02:51:15.566402 26677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:51:15.568568 26102 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 02:51:15.568833 26582 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:15.569221 26582 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:15.569415 26582 tablet_replica.cc:330] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: stopping tablet replica
W20260928 02:51:15.569476 26102 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 -> Peer 105043607d974614bb4289128ca69849 (127.19.139.189:33411): Couldn't send request to peer 105043607d974614bb4289128ca69849. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:51:15.570173 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.19.139.188@KRBTEST.COM; defaulting to no policy
I20260928 02:51:15.570401 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.188@KRBTEST.COM" created.
I20260928 02:51:15.584574 26582 master.cc:552] Master@127.19.139.189:33411 shutdown complete.
I20260928 02:51:15.584638 26582 master_runner.cc:305] Clearing existing system tablet
I20260928 02:51:15.584861 26582 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.584918 26582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.586041 26582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.586850 26684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.587072 26582 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.587203 26582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.587244 26582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
uuid: "105043607d974614bb4289128ca69849"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "40f555c3f85cb6ea664426da1905f965"
server_key_iv: "d5315d841180918ca9e6d83b90ba8165"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.188.keytab.
Entry for principal oryx/127.19.139.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.188.keytab.
I20260928 02:51:15.589260 26582 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:15.591046 26582 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:51:15.591121 26582 log.cc:1196] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:51:15.591388 26582 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Deleting consensus metadata
I20260928 02:51:15.591539 26582 master_runner.cc:315] Copying system tablet from 127.19.139.190:33543
I20260928 02:51:15.591763 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:35263
--webserver_interface=127.19.139.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.188.keytab
--principal=oryx/127.19.139.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:51:15.592626 26582 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: tablet copy: Beginning tablet copy session from remote peer at address 127.19.139.190:33543
I20260928 02:51:15.603296 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.593231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.19.139.190:33543)
0928 02:51:15.593357 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.593361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.594912 (+ 1551us) server_negotiation.cc:407] Connection header received
0928 02:51:15.595038 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.595041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.595097 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.595186 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.595696 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.596665 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.597843 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.598191 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.598307 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.599623 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.599627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.599630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.599661 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.601394 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.601896 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.601900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.601902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.602000 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.602276 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.602279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.602281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.602521 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.602621 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.603002 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.603132 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:51:15.603308 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.593130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:57550)
0928 02:51:15.594782 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.594798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.594928 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.595226 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.595229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.595435 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.595573 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.595581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.596787 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.596791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.597713 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.597723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.598314 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.598317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.598416 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.599013 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.599027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.599460 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.601530 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.601534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.601536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.601792 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.602093 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.602096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.602098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.602185 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.602674 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.602678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.602789 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.602917 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.603091 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"thread_start_us":86,"threads_started":1}
I20260928 02:51:15.604286 26138 tablet_copy_service.cc:140] P 7064d87b6446440eb11e70ad20437c70: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 105043607d974614bb4289128ca69849 ({username='oryx', principal='oryx/127.19.139.189@KRBTEST.COM'} at 127.0.0.1:57550)
I20260928 02:51:15.604491 26138 tablet_copy_service.cc:161] P 7064d87b6446440eb11e70ad20437c70: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 105043607d974614bb4289128ca69849 at {username='oryx', principal='oryx/127.19.139.189@KRBTEST.COM'} at 127.0.0.1:57550: session id = 105043607d974614bb4289128ca69849-00000000000000000000000000000000
I20260928 02:51:15.606338 26138 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:51:15.608007 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:51:15.610399 26582 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: tablet copy: Starting download of 0 data blocks...
I20260928 02:51:15.610605 26582 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: tablet copy: Starting download of 1 WAL segments...
I20260928 02:51:15.612501 26582 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:51:15.614560 26138 tablet_copy_service.cc:342] P 7064d87b6446440eb11e70ad20437c70: Request end of tablet copy session 105043607d974614bb4289128ca69849-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.139.189@KRBTEST.COM'} at 127.0.0.1:57550
I20260928 02:51:15.614696 26138 tablet_copy_service.cc:433] P 7064d87b6446440eb11e70ad20437c70: ending tablet copy session 105043607d974614bb4289128ca69849-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 105043607d974614bb4289128ca69849
I20260928 02:51:15.622644 26582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:15.624660 26692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:15.624706 26693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.628333 26582 server_base.cc:1034] running on GCE node
W20260928 02:51:15.628424 26695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.628696 26582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:15.629055 26582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:15.630188 26582 hybrid_clock.cc:648] HybridClock initialized: now 1790563875630178 us; error 34 us; skew 500 ppm
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:15.631798 26582 init.cc:377] Logged in from keytab as oryx/127.19.139.189@KRBTEST.COM (short username oryx)
I20260928 02:51:15.632344 26582 webserver.cc:460] Webserver started at http://127.19.139.189:41925/ using document root <none> and password file <none>
I20260928 02:51:15.632524 26582 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.632556 26582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.633893 26582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.634884 26702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.635140 26582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.635249 26582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
uuid: "105043607d974614bb4289128ca69849"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "40f555c3f85cb6ea664426da1905f965"
server_key_iv: "d5315d841180918ca9e6d83b90ba8165"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.635398 26582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.661999 26582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:15.664445 26582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:15.669636 26582 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.189:33411
I20260928 02:51:15.669667 26754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.189:33411 every 8 connection(s)
I20260928 02:51:15.670444 26582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
I20260928 02:51:15.672329 26755 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:51:15.673476 26755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Bootstrap starting.
I20260928 02:51:15.684638 26755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:51:15.684978 26755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Bootstrap complete.
I20260928 02:51:15.685634 26755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } }
I20260928 02:51:15.685789 26755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 105043607d974614bb4289128ca69849, State: Initialized, Role: LEARNER
I20260928 02:51:15.685966 26755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } }
I20260928 02:51:15.686319 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } }
I20260928 02:51:15.686542 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:51:15.686446 26755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:15.691123 26769 catalog_manager.cc:1259] Loaded cluster ID: 6db09412b7294458b363ab483943b76a
I20260928 02:51:15.691190 26769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: loading cluster ID for follower catalog manager: success
I20260928 02:51:15.693284 26769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: acquiring CA information for follower catalog manager: success
I20260928 02:51:15.694731 26769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:51:15.719875 26690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:15.720218 26690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:15.720276 26690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:15.724521 26690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:15.724586 26690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:15.724606 26690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:15.724622 26690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:15.724640 26690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:15.730321 26690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.188.keytab
--principal=oryx/127.19.139.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.139.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:15.731521 26690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:15.732646 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:15.741115 26777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:15.741115 26775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.741173 26690 server_base.cc:1034] running on GCE node
W20260928 02:51:15.741194 26774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.741959 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:15.743062 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:15.744235 26690 hybrid_clock.cc:648] HybridClock initialized: now 1790563875744212 us; error 34 us; skew 500 ppm
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:15.747025 26690 init.cc:377] Logged in from keytab as oryx/127.19.139.188@KRBTEST.COM (short username oryx)
I20260928 02:51:15.748005 26690 webserver.cc:460] Webserver started at http://127.19.139.188:43483/ using document root <none> and password file <none>
I20260928 02:51:15.748638 26690 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.748713 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.748919 26690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:15.750998 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/instance:
uuid: "6f670347ea08426694fe258964841247"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "daa4aac838604d44a90cea85b7ab7136"
server_key_iv: "11b4c9f89852baf99fca141ffd171ac9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.751557 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal/instance:
uuid: "6f670347ea08426694fe258964841247"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "daa4aac838604d44a90cea85b7ab7136"
server_key_iv: "11b4c9f89852baf99fca141ffd171ac9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.755617 26690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:15.758167 26784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.759510 26690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:15.759629 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
uuid: "6f670347ea08426694fe258964841247"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "daa4aac838604d44a90cea85b7ab7136"
server_key_iv: "11b4c9f89852baf99fca141ffd171ac9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.759728 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.793823 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:15.796919 26690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:15.797075 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:15.803929 26690 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.188:35263
I20260928 02:51:15.804206 26836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.188:35263 every 8 connection(s)
I20260928 02:51:15.805018 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
I20260928 02:51:15.807869 26837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:15.809698 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26690
I20260928 02:51:15.812552 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.139.188@KRBTEST.COM
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.188@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:15.824193 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.814716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.19.139.190:33543)
0928 02:51:15.814836 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.814841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.815118 (+ 277us) server_negotiation.cc:407] Connection header received
0928 02:51:15.815243 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.815246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.815296 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.815360 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.816031 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.816613 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.817503 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.817813 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.817901 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.820347 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.820351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.820353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.820382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.821935 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.822531 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.822535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.822536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.822616 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.822963 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.822967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.822968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.823184 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.823290 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.823822 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.824004 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:51:15.825106 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.811288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.19.139.188:35263)
0928 02:51:15.811728 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.811745 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.811789 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:51:15.812508 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.812528 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.812926 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.813298 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.814132 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.815557 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.816465 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.816971 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.817110 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.819811 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.819819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.819832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.819855 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.821896 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.822467 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.822473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.822475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.822625 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.822917 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.822920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.822922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.823403 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.823570 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.823821 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.824263 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260928 02:51:15.825105 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.814548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:57562)
0928 02:51:15.814998 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.815037 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.815156 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.815423 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.815429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.815776 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.815896 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.815902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.816735 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.816739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.817383 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.817392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.817951 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.817955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.818058 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.818543 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.818563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.820208 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.822044 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.822053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.822064 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.822429 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.822712 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.822715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.822717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.822870 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.823391 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.823397 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.823575 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.823759 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.824186 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":59,"thread_start_us":168,"threads_started":1}
I20260928 02:51:15.826304 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.188@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
I20260928 02:51:15.834198 26309 heartbeater.cc:503] Master 127.19.139.190:33543 was elected leader, sending a full tablet report...
I20260928 02:51:15.834641 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.826783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56662)
0928 02:51:15.826952 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.826966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.827041 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.827328 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.827331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.827505 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.827607 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.827612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.828527 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.828530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.829065 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.829072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.829602 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.829607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.829722 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.830244 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.830262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.831590 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.833221 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.833223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.833225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.833430 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.833687 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.833690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.833691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.833803 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.834140 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.834144 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.834230 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.834326 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.834469 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 02:51:15.834676 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.826880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.19.139.189:33411)
0928 02:51:15.827114 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.827117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.827128 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:51:15.827164 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.827167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.827214 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.827308 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.827701 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.828436 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.829175 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.829456 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.829556 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.831730 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.831733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.831736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.831760 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.833114 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.833515 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.833518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.833519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.833591 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.833872 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.833874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.833875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.834046 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.834131 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.834390 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.834532 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260928 02:51:15.835377 26837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.188@KRBTEST.COM for oryx/127.19.139.188@KRBTEST.COM
I20260928 02:51:15.843194 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.835908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32880)
0928 02:51:15.836038 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.836053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.836135 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.836376 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.836379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.836528 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.836598 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.836603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.837694 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.837696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.838201 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.838208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.838638 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.838641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.838709 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.839124 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.839138 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.840392 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.841737 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.841745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.841747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.841946 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.842267 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.842270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.842271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.842372 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.842744 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.842746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.842837 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.842924 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.843034 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:51:15.843204 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.835946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.19.139.188:35263)
0928 02:51:15.836039 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.836043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.836096 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:51:15.836230 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.836233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.836288 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.836357 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.836691 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.837228 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.838302 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.838549 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.838632 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.840507 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.840510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.840512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.840533 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.841641 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.842050 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.842053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.842054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.842132 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.842481 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.842484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.842485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.842670 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.842746 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.842986 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.843080 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:51:15.846125 26837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Bootstrap starting.
I20260928 02:51:15.848459 26837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:15.849169 26837 log.cc:824] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:15.851065 26837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: No bootstrap required, opened a new log
I20260928 02:51:15.854214 26837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } }
I20260928 02:51:15.854463 26837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:15.854525 26837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f670347ea08426694fe258964841247, State: Initialized, Role: FOLLOWER
I20260928 02:51:15.854962 26837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } }
I20260928 02:51:15.855507 26843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } } }
I20260928 02:51:15.855634 26843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:15.856282 26837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:51:15.860850 26854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:51:15.860953 26854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:51:15.864799 26690 master_runner.cc:428] Detected that this master 6f670347ea08426694fe258964841247 is joining an existing cluster
I20260928 02:51:15.864868 26690 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.139.188:35263
I20260928 02:51:15.878772 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26180
I20260928 02:51:15.884188 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:45839
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:33543
--webserver_port=39851
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:51:15.892956 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.867086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:57566)
0928 02:51:15.867570 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.867585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.867753 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.868100 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.868102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.868367 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.868556 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.868563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.869664 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.869667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.870848 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.870855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.871500 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.871503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.871587 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.872288 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.872301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.888962 (+ 16661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.891347 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.891352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.891572 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.891852 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.891856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.891944 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.892390 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.892393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.892529 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.892621 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.892797 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":19,"thread_start_us":128,"threads_started":1}
I20260928 02:51:15.893607 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.867151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.19.139.190:33543)
0928 02:51:15.867281 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.867286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.867665 (+ 379us) server_negotiation.cc:407] Connection header received
0928 02:51:15.867872 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.867875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.867923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.867967 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.868684 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.869532 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.870958 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.871288 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.871359 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.889218 (+ 17859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.889223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.889226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.889263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.891230 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891676 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.891679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.891681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.891763 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.892036 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.892039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.892041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.892262 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.892347 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.893333 (+ 986us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.893462 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:51:15.895349 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.867715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.19.139.189:33411)
0928 02:51:15.867835 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.867840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.868529 (+ 689us) server_negotiation.cc:407] Connection header received
0928 02:51:15.868699 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.868702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.868753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.868805 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.869438 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.870250 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.871502 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.871871 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.871950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.889286 (+ 17336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.889290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.889293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.889322 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.891019 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891566 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.891570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.891572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.891661 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891968 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.891972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.891974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.892262 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.892336 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.895074 (+ 2738us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.895207 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:15.895771 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.867604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56678)
0928 02:51:15.868449 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.868462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.868591 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.868963 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.868966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.869163 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.869281 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.869288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.870406 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.870409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.871350 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.871361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.872098 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.872101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.872225 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.872616 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.872629 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.889124 (+ 16495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.891168 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.891175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.891437 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.891770 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.891772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.891774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.891856 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.892399 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.892401 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.894195 (+ 1794us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.895472 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.895610 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":30,"thread_start_us":84,"threads_started":1}
I20260928 02:51:15.898653 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.867832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.19.139.188:35263)
0928 02:51:15.867938 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.867942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.868210 (+ 268us) server_negotiation.cc:407] Connection header received
0928 02:51:15.868355 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.868358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.868401 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.868467 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.868985 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.869668 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.870838 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.871116 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.871178 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.892646 (+ 21468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.892650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.892652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.892671 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.894817 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.897493 (+ 2676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.897497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.897498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.897576 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.897825 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.897828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.897830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.898026 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.898104 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.898364 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.898476 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":44}
I20260928 02:51:15.898664 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.867295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32882)
0928 02:51:15.868100 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.868113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.868257 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.868557 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.868559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.868770 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.868877 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.868883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.869763 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.869765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.870727 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.870737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.871333 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.871336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.871412 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.871912 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.871926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.892516 (+ 20590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.896684 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.896687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.896689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.897404 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.897666 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.897668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.897670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.897745 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.898144 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.898147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.898215 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.898285 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.898426 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":30,"thread_start_us":82,"threads_started":1}
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563874, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.190@KRBTEST.COM for oryx/127.19.139.188@KRBTEST.COM
I20260928 02:51:15.906134 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.897924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32890)
0928 02:51:15.898067 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.898082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.898165 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.899036 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.899040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.899221 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.899332 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.899339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.900286 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.900290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.900833 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.900841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.901266 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.901269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.901368 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.901792 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.901806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.903034 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.904719 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.904722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.904725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.904914 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.905161 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.905164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.905166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.905246 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.905610 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.905613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.905709 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.905796 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.905964 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:51:15.906134 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.898305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.19.139.188:35263)
0928 02:51:15.898811 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.898815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.898826 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:51:15.898893 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.898896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.898942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.899002 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.899460 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.900098 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.900947 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.901165 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.901249 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.903203 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.903206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.903210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.903235 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.904615 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.905002 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.905005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.905007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.905083 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.905325 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.905329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.905330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.905521 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.905603 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.905857 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.905963 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":119,"threads_started":1}
I20260928 02:51:15.907404 26118 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.139.188:35263: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6f670347ea08426694fe258964841247"
member_type: NON_VOTER
last_known_addr {
host: "127.19.139.188"
port: 35263
}
attrs {
promote: true
}
}
dest_uuid: "7064d87b6446440eb11e70ad20437c70"
cas_config_opid_index: 5
I20260928 02:51:15.907969 26118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } }
I20260928 02:51:15.909137 26678 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6f670347ea08426694fe258964841247 (127.19.139.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.910181 26811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:15.910596 26678 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.139.188:35263
I20260928 02:51:15.910759 26678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.910842 26678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:15.911540 26690 master.cc:537] Master@127.19.139.188:35263 shutting down...
I20260928 02:51:15.911588 26811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Refusing update from remote peer 7064d87b6446440eb11e70ad20437c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:51:15.913226 26679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:51:15.917032 26105 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 -> Peer 6f670347ea08426694fe258964841247 (127.19.139.188:35263): Couldn't send request to peer 6f670347ea08426694fe258964841247. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:51:15.917445 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.909187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56680)
0928 02:51:15.913022 (+ 3835us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.913036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.913121 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.913563 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.913567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.913584 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:15.913677 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.913684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.914682 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.914686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.915853 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.915863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.917027 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.917030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.917122 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.917169 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.917237 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.917316 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3746}
I20260928 02:51:15.917855 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.909436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.19.139.189:33411)
0928 02:51:15.912908 (+ 3472us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.912913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.913207 (+ 294us) server_negotiation.cc:407] Connection header received
0928 02:51:15.913267 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.913270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.913336 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.913425 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:15.913808 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.914558 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.915993 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.916705 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.916817 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.916955 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.917585 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.917731 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3365}
I20260928 02:51:15.918458 26729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 LEARNER]: Refusing update from remote peer 7064d87b6446440eb11e70ad20437c70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:51:15.918649 26690 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:15.918943 26690 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:15.919070 26690 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: stopping tablet replica
I20260928 02:51:15.919035 26679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:51:15.920883 26729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6f670347ea08426694fe258964841247 (127.19.139.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.922356 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.922472 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:51:15.922765 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7064d87b6446440eb11e70ad20437c70. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: NON_VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: true } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.922837 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:51:15.929548 26679 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: attempting to promote NON_VOTER 105043607d974614bb4289128ca69849 to VOTER
I20260928 02:51:15.930011 26679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } }
I20260928 02:51:15.931317 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.931023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32902)
0928 02:51:15.931175 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.931245 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:35263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:51:15.931865 26729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 LEARNER]: Refusing update from remote peer 7064d87b6446440eb11e70ad20437c70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:51:15.932255 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.931802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32906)
0928 02:51:15.932116 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.932191 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:35263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260928 02:51:15.932375 26678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 02:51:15.932654 26105 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 -> Peer 6f670347ea08426694fe258964841247 (127.19.139.188:35263): Couldn't send request to peer 6f670347ea08426694fe258964841247. Status: Network error: Client connection negotiation failed: client connection to 127.19.139.188:35263: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:51:15.933828 26677 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 105043607d974614bb4289128ca69849 (127.19.139.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.935098 26677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.935201 26677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:15.935318 26677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.935390 26677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:15.936309 26690 master.cc:552] Master@127.19.139.188:35263 shutdown complete.
I20260928 02:51:15.936352 26690 master_runner.cc:305] Clearing existing system tablet
I20260928 02:51:15.936576 26690 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.936626 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.937781 26690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.938110 26729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 105043607d974614bb4289128ca69849 (127.19.139.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.939440 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:15.939564 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:15.940317 26868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.940538 26690 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.940667 26690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:15.940725 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
uuid: "6f670347ea08426694fe258964841247"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "daa4aac838604d44a90cea85b7ab7136"
server_key_iv: "11b4c9f89852baf99fca141ffd171ac9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.942339 26690 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:15.944252 26690 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:51:15.944336 26690 log.cc:1196] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:51:15.944581 26690 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Deleting consensus metadata
I20260928 02:51:15.944727 26690 master_runner.cc:315] Copying system tablet from 127.19.139.190:33543
I20260928 02:51:15.945737 26690 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: tablet copy: Beginning tablet copy session from remote peer at address 127.19.139.190:33543
I20260928 02:51:15.953104 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.946333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.19.139.190:33543)
0928 02:51:15.946458 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:15.946463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:15.946623 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:51:15.946742 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:15.946745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:15.946794 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:15.946862 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:15.947335 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.947872 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.948756 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.948979 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.949070 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.950236 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:15.950239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:15.950242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:15.950269 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:15.951546 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.951932 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.951936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.951938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.952017 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:15.952307 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:15.952310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:15.952312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:15.952499 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:15.952584 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:15.952835 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:51:15.952942 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:51:15.953274 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:15.946239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:57568)
0928 02:51:15.946532 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:15.946545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:15.946658 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:15.946887 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:15.946890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:15.947091 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:15.947202 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.947208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.947991 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.947995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.948638 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:15.948648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:15.949084 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:15.949087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:15.949238 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:15.949755 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:15.949770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:15.950095 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:15.951646 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.951649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.951651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.951844 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.952097 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:15.952099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:15.952102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:15.952217 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:15.952586 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:15.952589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:15.952686 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 02:51:15.952774 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:51:15.952919 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":86,"threads_started":1}
I20260928 02:51:15.953786 26138 tablet_copy_service.cc:140] P 7064d87b6446440eb11e70ad20437c70: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6f670347ea08426694fe258964841247 ({username='oryx', principal='oryx/127.19.139.188@KRBTEST.COM'} at 127.0.0.1:57568)
I20260928 02:51:15.953886 26138 tablet_copy_service.cc:161] P 7064d87b6446440eb11e70ad20437c70: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6f670347ea08426694fe258964841247 at {username='oryx', principal='oryx/127.19.139.188@KRBTEST.COM'} at 127.0.0.1:57568: session id = 6f670347ea08426694fe258964841247-00000000000000000000000000000000
I20260928 02:51:15.954696 26138 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:51:15.955845 26690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:51:15.958142 26690 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: tablet copy: Starting download of 0 data blocks...
I20260928 02:51:15.958328 26690 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: tablet copy: Starting download of 1 WAL segments...
I20260928 02:51:15.959748 26690 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:51:15.961443 26138 tablet_copy_service.cc:342] P 7064d87b6446440eb11e70ad20437c70: Request end of tablet copy session 6f670347ea08426694fe258964841247-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.139.188@KRBTEST.COM'} at 127.0.0.1:57568
I20260928 02:51:15.961529 26138 tablet_copy_service.cc:433] P 7064d87b6446440eb11e70ad20437c70: ending tablet copy session 6f670347ea08426694fe258964841247-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6f670347ea08426694fe258964841247
I20260928 02:51:15.968564 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:15.970163 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:15.970458 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.970674 26690 server_base.cc:1034] running on GCE node
W20260928 02:51:15.970346 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:15.970902 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:15.971200 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:15.972326 26690 hybrid_clock.cc:648] HybridClock initialized: now 1790563875972313 us; error 32 us; skew 500 ppm
Sep 28 02:51:15 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563875, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:15.973829 26690 init.cc:377] Logged in from keytab as oryx/127.19.139.188@KRBTEST.COM (short username oryx)
I20260928 02:51:15.974282 26690 webserver.cc:460] Webserver started at http://127.19.139.188:42329/ using document root <none> and password file <none>
I20260928 02:51:15.974460 26690 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:15.974501 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:15.975611 26690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.976522 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:15.976725 26690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:15.976796 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
uuid: "6f670347ea08426694fe258964841247"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "daa4aac838604d44a90cea85b7ab7136"
server_key_iv: "11b4c9f89852baf99fca141ffd171ac9"
server_key_version: "encryptionkey@0"
I20260928 02:51:15.976881 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:15.995976 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:15.997843 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:16.002542 26690 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.188:35263
I20260928 02:51:16.002554 26937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.188:35263 every 8 connection(s)
I20260928 02:51:16.003074 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
I20260928 02:51:16.011392 26938 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:51:16.012398 26938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Bootstrap starting.
I20260928 02:51:16.021601 26938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:51:16.021900 26938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: Bootstrap complete.
I20260928 02:51:16.022450 26938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } }
I20260928 02:51:16.022588 26938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f670347ea08426694fe258964841247, State: Initialized, Role: LEARNER
I20260928 02:51:16.022697 26938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } }
I20260928 02:51:16.023051 26941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:16.023187 26938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:16.023254 26941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:51:16.026264 26952 catalog_manager.cc:1259] Loaded cluster ID: 6db09412b7294458b363ab483943b76a
I20260928 02:51:16.026393 26952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: loading cluster ID for follower catalog manager: success
I20260928 02:51:16.028383 26952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: acquiring CA information for follower catalog manager: success
I20260928 02:51:16.029558 26952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:51:16.045351 26863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:16.045742 26863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:16.045805 26863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:16.051980 26863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:16.052079 26863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:16.052183 26863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:16.057912 26443 heartbeater.cc:503] Master 127.19.139.190:33543 was elected leader, sending a full tablet report...
I20260928 02:51:16.058672 26863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:45839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=39851
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:16.059798 26863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:16.060938 26863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:16.070961 26957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:16.071049 26863 server_base.cc:1034] running on GCE node
W20260928 02:51:16.070938 26960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:16.071046 26958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:16.071728 26863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:16.072454 26863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:16.073622 26863 hybrid_clock.cc:648] HybridClock initialized: now 1790563876073600 us; error 35 us; skew 500 ppm
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:16.076494 26863 init.cc:377] Logged in from keytab as oryx/127.19.139.129@KRBTEST.COM (short username oryx)
I20260928 02:51:16.077473 26863 webserver.cc:460] Webserver started at http://127.19.139.129:39851/ using document root <none> and password file <none>
I20260928 02:51:16.078101 26863 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:16.078186 26863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:16.082296 26863 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:16.084520 26967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.085830 26863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:51:16.085937 26863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "9224ae22ca9c41dcbabe9026ec7699aa"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "6a54ca76da86f2f43f95ae5e3a3bea14"
server_key_iv: "3e77871b55bd64cac4b25e25de66c570"
server_key_version: "encryptionkey@0"
I20260928 02:51:16.086601 26863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:16.096557 26863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:16.100049 26863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:16.100275 26863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:16.100899 26863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:16.101811 26863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:16.101876 26863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.101934 26863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:16.101958 26863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.110514 26863 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:45839
I20260928 02:51:16.110587 27080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:45839 every 8 connection(s)
I20260928 02:51:16.111544 26863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:16.111785 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 26863
I20260928 02:51:16.112001 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26314
I20260928 02:51:16.118197 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:41777
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:33543
--webserver_port=42185
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.188@KRBTEST.COM
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
I20260928 02:51:16.129354 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.115592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:59941 (local address 127.19.139.190:33543)
0928 02:51:16.115705 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.115710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.116840 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 02:51:16.117001 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.117004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.117054 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.117104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.118332 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.119180 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.120730 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.121085 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.121182 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.125430 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.125434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.125437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.125469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.127284 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127772 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.127776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.127778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.127896 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.128184 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.128188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.128189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.128396 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.128467 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.129051 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.129179 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:16.129361 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.115185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:39327 (local address 127.19.139.189:33411)
0928 02:51:16.115313 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.115317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.115554 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:51:16.116590 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.116594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.116649 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.116731 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.118495 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.119245 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.121660 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.121984 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.122065 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.125399 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.125403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.125406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.125438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.127160 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127708 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.127712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.127713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.127804 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.128112 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.128115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.128117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.128367 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.128456 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.129065 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.129195 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:51:16.130234 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.114752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:50165 (local address 127.19.139.188:35263)
0928 02:51:16.116338 (+ 1586us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.116343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.116358 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:51:16.116853 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.116857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.116946 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.117027 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.118356 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.119323 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.120838 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.121183 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.121267 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.124883 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.124888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.124891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.124924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.126830 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127474 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.127477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.127479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.127563 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127957 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.127960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.127962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.128245 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.128339 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.129946 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.130075 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1509,"thread_start_us":92,"threads_started":1}
I20260928 02:51:16.130904 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.114199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.19.139.129:39327)
0928 02:51:16.115411 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.115467 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.116366 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.116929 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.116937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.117489 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.118378 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.118384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.119525 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.119528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.121530 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.121540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.122238 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.122241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.122348 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.122911 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.122926 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.125216 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.127292 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.127297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.127584 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.127914 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.127918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.128005 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.128527 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.128531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.128767 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.129599 (+ 832us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.129807 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":1564,"spinlock_wait_cycles":23040,"thread_start_us":114,"threads_started":1}
I20260928 02:51:16.130903 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.115265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:59941)
0928 02:51:16.116749 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.116761 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.116903 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.117242 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.117245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.117520 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.118199 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.118214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.119797 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.119799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.120620 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.120629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.121357 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.121362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.121646 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.122464 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.122482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.125263 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.127401 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.127407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.127667 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.127994 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.127999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.128069 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.128515 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.128517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.128764 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.129545 (+ 781us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.129735 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"mutex_wait_us":159,"spinlock_wait_cycles":40832,"thread_start_us":82,"threads_started":1}
I20260928 02:51:16.131110 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.114200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.19.139.129:50165)
0928 02:51:16.115508 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.115521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.116369 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.117150 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.117153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.117504 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.118201 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.118216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.119504 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.119508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.120718 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.120729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.121383 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.121386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.121651 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.122450 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.122473 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.124685 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.126970 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.126977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.126997 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.127368 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.127662 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.127665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.127667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.127851 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.128393 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.128400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.128764 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.128975 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.129394 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":229,"spinlock_wait_cycles":69632,"thread_start_us":139,"threads_started":1}
I20260928 02:51:16.132637 27082 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33543
I20260928 02:51:16.132874 27083 heartbeater.cc:347] Connected to a master server at 127.19.139.189:33411
I20260928 02:51:16.132889 27081 heartbeater.cc:347] Connected to a master server at 127.19.139.188:35263
I20260928 02:51:16.132965 27083 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.132977 27081 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.133463 27081 heartbeater.cc:511] Master 127.19.139.188:35263 requested a full tablet report, sending...
I20260928 02:51:16.133464 27083 heartbeater.cc:511] Master 127.19.139.189:33411 requested a full tablet report, sending...
I20260928 02:51:16.132887 27082 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.134371 27082 heartbeater.cc:511] Master 127.19.139.190:33543 requested a full tablet report, sending...
I20260928 02:51:16.135076 26118 ts_manager.cc:194] Re-registered known tserver with Master: 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839)
I20260928 02:51:16.135082 26894 ts_manager.cc:194] Registered new tserver with Master: 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839)
I20260928 02:51:16.135761 26118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:59941
I20260928 02:51:16.136094 26719 ts_manager.cc:194] Registered new tserver with Master: 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839)
W20260928 02:51:16.236354 27089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:16.236672 27089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:16.236733 27089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:16.240832 27089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:16.240901 27089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:16.240993 27089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:16.246371 27089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:41777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=42185
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:16.247506 27089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:16.248672 27089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:16.258770 27095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:16.258834 27094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:16.258873 27097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:16.258834 27089 server_base.cc:1034] running on GCE node
I20260928 02:51:16.259512 27089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:16.260207 27089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:16.261413 27089 hybrid_clock.cc:648] HybridClock initialized: now 1790563876261411 us; error 56 us; skew 500 ppm
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:16.264113 27089 init.cc:377] Logged in from keytab as oryx/127.19.139.130@KRBTEST.COM (short username oryx)
I20260928 02:51:16.265070 27089 webserver.cc:460] Webserver started at http://127.19.139.130:42185/ using document root <none> and password file <none>
I20260928 02:51:16.265648 27089 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:16.265722 27089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:16.269677 27089 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:16.271842 27104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.273375 27089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:16.273929 26577 heartbeater.cc:503] Master 127.19.139.190:33543 was elected leader, sending a full tablet report...
I20260928 02:51:16.273517 27089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "bc8630532d7243439983b8cff12fa436"
format_stamp: "Formatted at 2026-09-28 02:51:14 on dist-test-slave-5vc0"
server_key: "735ea0cb2f85e8f56f7ccb071cc05cea"
server_key_iv: "de7c4283606eed6e7368d405f4c18626"
server_key_version: "encryptionkey@0"
I20260928 02:51:16.274158 27089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:16.293864 27089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:16.296964 27089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:16.297142 27089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:16.297741 27089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:16.298611 27089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:16.298657 27089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.298710 27089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:16.298736 27089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.308333 27089 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:41777
I20260928 02:51:16.308588 27217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:41777 every 8 connection(s)
I20260928 02:51:16.309669 27089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:16.316419 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 27089
I20260928 02:51:16.316598 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26448
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.188@KRBTEST.COM
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
I20260928 02:51:16.323974 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:46441
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:33543
--webserver_port=37863
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:51:16.325604 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.313057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:59821 (local address 127.19.139.190:33543)
0928 02:51:16.313172 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.313177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.313214 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:51:16.313670 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.313674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.313729 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.313782 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.315173 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.315703 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.317640 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.317982 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.318064 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.321827 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.321831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.321834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.321861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.323468 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.324496 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.324499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.324501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.324577 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.324809 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.324812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.324813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.324991 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.325053 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.325307 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.325439 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:16.326612 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.312931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:59821)
0928 02:51:16.313147 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.313159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.313456 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.313941 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.313949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.314432 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.315013 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.315031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.315839 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.315841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.317510 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.317519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.318208 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.318211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.318299 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.318851 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.318869 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.321684 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.323642 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.323645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.323647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.324393 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.324667 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.324670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.324671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.324736 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.325099 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.325101 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.325174 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.325807 (+ 633us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.325957 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":449,"spinlock_wait_cycles":37376,"thread_start_us":73,"threads_started":1}
I20260928 02:51:16.327037 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.311454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:53745 (local address 127.19.139.188:35263)
0928 02:51:16.311583 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.311588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.312556 (+ 968us) server_negotiation.cc:407] Connection header received
0928 02:51:16.313640 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.313643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.313700 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.313771 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.315152 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.315976 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.317412 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.317752 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.317838 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.321002 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.321006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.321009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.321039 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.322961 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.323553 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.323557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.323559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.323648 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.324052 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.324055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.324057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.324309 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.324395 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.326745 (+ 2350us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.326891 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:51:16.326619 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.19.139.130:53745)
0928 02:51:16.312395 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.312448 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.313450 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.313939 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.313947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.314433 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.315015 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.315030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.316105 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.316109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.317281 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.317291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.318002 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.318006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.318196 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.319107 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.319122 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.320803 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.323087 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.323093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.323111 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.323429 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.323744 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.323747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.323749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.323932 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.324449 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.324456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.324804 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.325048 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.325442 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":84,"spinlock_wait_cycles":8064,"thread_start_us":128,"threads_started":1}
I20260928 02:51:16.327731 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.19.139.130:36537)
0928 02:51:16.312396 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.312448 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.313449 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.313941 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.313949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.314432 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.315013 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.315031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.317062 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.317065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.317860 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.317868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.318541 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.318543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.318615 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.318969 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.318982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.321611 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.323660 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.323662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.323664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.323929 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.326507 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.326509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.326511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.326603 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.327129 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.327131 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.327215 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.327435 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.327579 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":57,"spinlock_wait_cycles":99712,"thread_start_us":187,"threads_started":1}
I20260928 02:51:16.327843 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.311451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:36537 (local address 127.19.139.189:33411)
0928 02:51:16.311586 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.311590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.312557 (+ 967us) server_negotiation.cc:407] Connection header received
0928 02:51:16.313685 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.313688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.313740 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.313786 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.315183 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.315976 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.317982 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.318298 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.318370 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.321771 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.321775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.321778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.321806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.323564 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.324050 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.324054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.324056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.324190 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.326722 (+ 2532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.326726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.326728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.326977 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.327062 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.327589 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.327705 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:51:16.328718 27219 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33543
I20260928 02:51:16.328774 27220 heartbeater.cc:347] Connected to a master server at 127.19.139.189:33411
I20260928 02:51:16.328976 27220 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.329454 27218 heartbeater.cc:347] Connected to a master server at 127.19.139.188:35263
I20260928 02:51:16.329533 27218 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.329555 27220 heartbeater.cc:511] Master 127.19.139.189:33411 requested a full tablet report, sending...
I20260928 02:51:16.329670 27218 heartbeater.cc:511] Master 127.19.139.188:35263 requested a full tablet report, sending...
I20260928 02:51:16.328977 27219 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.330204 27219 heartbeater.cc:511] Master 127.19.139.190:33543 requested a full tablet report, sending...
I20260928 02:51:16.330708 26894 ts_manager.cc:194] Registered new tserver with Master: bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777)
I20260928 02:51:16.330726 26719 ts_manager.cc:194] Registered new tserver with Master: bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777)
I20260928 02:51:16.331066 26118 ts_manager.cc:194] Re-registered known tserver with Master: bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777)
I20260928 02:51:16.331755 26118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:59821
W20260928 02:51:16.439918 27225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:16.440292 27225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:16.440356 27225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:16.444481 27225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:16.444540 27225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:16.444605 27225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:16.450012 27225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:46441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=37863
--tserver_master_addrs=127.19.139.190:33543,127.19.139.189:33411,127.19.139.188:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:16.451085 27225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:16.452288 27225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:16.462751 27231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:16.462765 27232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:16.463011 27225 server_base.cc:1034] running on GCE node
W20260928 02:51:16.462742 27234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:16.463557 27225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:16.464246 27225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:16.465396 27225 hybrid_clock.cc:648] HybridClock initialized: now 1790563876465378 us; error 40 us; skew 500 ppm
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:16.468225 27225 init.cc:377] Logged in from keytab as oryx/127.19.139.131@KRBTEST.COM (short username oryx)
I20260928 02:51:16.469182 27225 webserver.cc:460] Webserver started at http://127.19.139.131:37863/ using document root <none> and password file <none>
I20260928 02:51:16.469748 27225 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:16.469822 27225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:16.474124 27225 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:16.476385 27241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.477917 27225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:51:16.478020 27225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "38f75d9c849745b999b55d607315f5a3"
format_stamp: "Formatted at 2026-09-28 02:51:15 on dist-test-slave-5vc0"
server_key: "af608b1b2bbd2b3b2ef194432fc97bda"
server_key_iv: "8d59ede06dc38b05c1a57ab2b773f7c9"
server_key_version: "encryptionkey@0"
I20260928 02:51:16.478609 27225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:16.490525 27225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:16.492336 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.488118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.19.139.188:35263)
0928 02:51:16.488302 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.488307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.488442 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:51:16.488598 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.488602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.488665 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.488776 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:16.489081 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.489872 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.491135 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.491742 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.491868 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.492037 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.492102 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.492223 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:51:16.492341 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.488076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32914)
0928 02:51:16.488353 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.488367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.488492 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.488832 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.488836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.488872 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:16.488971 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.488976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.490007 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.490010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.491018 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.491030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.491885 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.491889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.491992 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.492021 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.492086 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.492175 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260928 02:51:16.493214 26912 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:51:16.493618 26941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7064d87b6446440eb11e70ad20437c70. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:16.493755 26941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:51:16.494370 27225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:16.494529 27225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:16.495098 27225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:16.495934 27225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:16.495978 27225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.496033 27225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:16.496062 27225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:16.504576 27225 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46441
I20260928 02:51:16.504624 27355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46441 every 8 connection(s)
I20260928 02:51:16.505614 27225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:16.512481 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 27225
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.189@KRBTEST.COM
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.188@KRBTEST.COM
Sep 28 02:51:16 dist-test-slave-5vc0 krb5kdc[26073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563876, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:16.523990 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.507495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:33855 (local address 127.19.139.188:35263)
0928 02:51:16.507631 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.507636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.508832 (+ 1196us) server_negotiation.cc:407] Connection header received
0928 02:51:16.509957 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.509961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.510018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.510093 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.511496 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.512036 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.514545 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.514823 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.514919 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.517689 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.517693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.517696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.517727 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.522113 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.522759 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.522762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.522764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.522842 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.523122 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.523126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.523128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.523343 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.523415 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.523722 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.523834 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:51:16.524088 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.507529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:49417 (local address 127.19.139.189:33411)
0928 02:51:16.507632 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.507636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.508832 (+ 1196us) server_negotiation.cc:407] Connection header received
0928 02:51:16.510212 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.510216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.510268 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.510353 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.511433 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.512265 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.513849 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.514088 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.514176 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.518025 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.518029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.518031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.518061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.519929 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.520627 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.520631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.520633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.520721 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.521805 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.521809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.521810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.522057 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.522140 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.523798 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.523922 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":44}
I20260928 02:51:16.524960 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.508027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.19.139.131:33855)
0928 02:51:16.508651 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.508715 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.509744 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.510290 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.510298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.510755 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.511265 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.511278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.512206 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.512210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.513710 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.513720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.515040 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.515044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.515128 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.515502 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.515516 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.517455 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.522427 (+ 4972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.522429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.522431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.522656 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.522945 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.522948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.522950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.523028 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.523479 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.523481 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.523560 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.524647 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.524789 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":284,"spinlock_wait_cycles":69248,"thread_start_us":149,"threads_started":1}
I20260928 02:51:16.524977 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.509182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:33627)
0928 02:51:16.509409 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.509420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.509740 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.510289 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.510298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.510744 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.511263 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.511276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.512388 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.512391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.513862 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.513869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.514572 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.514575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.515007 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.515613 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.515628 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.517839 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.521711 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.521714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.521716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.521941 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.522422 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.522424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.522426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.522553 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.522995 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.522998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.523079 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.523187 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.523557 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":393,"spinlock_wait_cycles":9088,"thread_start_us":71,"threads_started":1}
I20260928 02:51:16.525360 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.509302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:33627 (local address 127.19.139.190:33543)
0928 02:51:16.509439 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.509444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.509461 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:51:16.509968 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.509972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.510019 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.510071 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.511411 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.512264 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.513981 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.514282 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.514360 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.518195 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.518198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.518201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.518228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.521608 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.522226 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.522230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.522232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.522319 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.522657 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.522660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.522662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.522881 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.522958 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.525108 (+ 2150us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.525220 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:51:16.526038 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.508026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.19.139.131:49417)
0928 02:51:16.508651 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.508715 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.509738 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.510383 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.510385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.510738 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:16.511265 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.511278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.512388 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.512391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.513699 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.513711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.514231 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.514235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.514395 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.515015 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:16.515036 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:16.517866 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:16.520047 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.520053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.520072 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.520508 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.520824 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:16.520827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:16.520829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:16.521023 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:16.522210 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:16.522217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:16.522598 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.524282 (+ 1684us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.524474 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":73,"spinlock_wait_cycles":18816,"thread_start_us":140,"threads_started":1}
I20260928 02:51:16.526783 27357 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33543
I20260928 02:51:16.527070 27357 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.527657 27358 heartbeater.cc:347] Connected to a master server at 127.19.139.189:33411
I20260928 02:51:16.527707 27357 heartbeater.cc:511] Master 127.19.139.190:33543 requested a full tablet report, sending...
I20260928 02:51:16.527733 27358 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.528576 27358 heartbeater.cc:511] Master 127.19.139.189:33411 requested a full tablet report, sending...
I20260928 02:51:16.528944 26118 ts_manager.cc:194] Re-registered known tserver with Master: 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441)
I20260928 02:51:16.529232 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.515935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.19.139.189:33411)
0928 02:51:16.516338 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.516342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.518741 (+ 2399us) server_negotiation.cc:407] Connection header received
0928 02:51:16.518783 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.518785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.518829 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.518907 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.519588 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.520388 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.521497 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.521815 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.521917 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.525391 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.525395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.525398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.525435 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.527139 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.527607 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.527611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.527612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.527706 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.528006 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.528009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.528010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.528248 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.528366 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.528927 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.529054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":317,"thread_start_us":87,"threads_started":1}
I20260928 02:51:16.529541 26719 ts_manager.cc:194] Registered new tserver with Master: 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441)
I20260928 02:51:16.529637 26118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:33627
I20260928 02:51:16.529901 27356 heartbeater.cc:347] Connected to a master server at 127.19.139.188:35263
I20260928 02:51:16.529987 27356 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:16.530136 27356 heartbeater.cc:511] Master 127.19.139.188:35263 requested a full tablet report, sending...
I20260928 02:51:16.530812 26894 ts_manager.cc:194] Registered new tserver with Master: 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441)
I20260928 02:51:16.530967 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.515270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.19.139.190:33543)
0928 02:51:16.516532 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.516544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.516775 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:51:16.516904 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.516906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.516949 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.517023 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.517839 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.518641 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.519819 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.520175 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.520267 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.521829 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.521833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.521835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.521859 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.523640 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.524612 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.524615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.524617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.524700 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.526095 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.526099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.526100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.526314 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.526436 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.530650 (+ 4214us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.530793 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":75,"threads_started":1}
I20260928 02:51:16.531704 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.514522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.19.139.188:35263)
0928 02:51:16.518095 (+ 3573us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.518107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.518119 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:51:16.518151 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.518154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.518196 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.518280 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.522838 (+ 4558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.524842 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.526301 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.526643 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.526726 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.528248 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.528252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.528255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.528277 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.529779 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.530270 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.530273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.530274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.530365 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.530626 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.530629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.530630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.530840 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.530954 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.531493 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.531592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3502,"thread_start_us":86,"threads_started":1}
I20260928 02:51:16.535184 26118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:16.538507 26118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:16.541219 27229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: NON_VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: true } } }
I20260928 02:51:16.541369 27229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:16.544672 27376 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: attempting to promote NON_VOTER 6f670347ea08426694fe258964841247 to VOTER
W20260928 02:51:16.544942 27374 consensus_queue.cc:1159] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "105043607d974614bb4289128ca69849" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20260928 02:51:16.545197 27376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:16.546558 26912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 LEARNER]: Refusing update from remote peer 7064d87b6446440eb11e70ad20437c70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 02:51:16.547034 27229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 02:51:16.548318 26729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Refusing update from remote peer 7064d87b6446440eb11e70ad20437c70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 02:51:16.548807 27376 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 02:51:16.548975 27375 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6f670347ea08426694fe258964841247 (127.19.139.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.549378 26912 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6f670347ea08426694fe258964841247 (127.19.139.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.550007 26729 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6f670347ea08426694fe258964841247 (127.19.139.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.550482 27375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.550590 27375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:16.550694 27375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.550761 27375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:16.551644 26941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.551755 26941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:16.552635 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.552733 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:16.556717 27384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7064d87b6446440eb11e70ad20437c70" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:16.556845 27384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:16.561291 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.554909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46441 (local address 127.0.0.1:50042)
0928 02:51:16.555682 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.555694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.555777 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.556761 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.556764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.556780 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:16.556877 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.556883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.558294 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.558298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.559546 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.559554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.560873 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.560877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.560990 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.561013 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.561075 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.561146 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260928 02:51:16.561832 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.555072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.19.139.131:46441)
0928 02:51:16.555420 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.555431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.555859 (+ 428us) server_negotiation.cc:407] Connection header received
0928 02:51:16.555938 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.555943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.556459 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.556622 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:16.556987 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.558171 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.559676 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.560576 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.560687 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.560815 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.561469 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.561580 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
I20260928 02:51:16.562629 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.554867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:41777 (local address 127.0.0.1:57790)
0928 02:51:16.555177 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.555192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.555270 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.558290 (+ 3020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.558293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.558309 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:16.558383 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.558389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.559793 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.559797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.560970 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.560978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.562265 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.562269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.562360 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.562383 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.562430 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.562506 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":46}
I20260928 02:51:16.563321 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.555472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.19.139.130:41777)
0928 02:51:16.557102 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.557114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.557138 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:51:16.557214 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.557219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.557439 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.558516 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:16.558652 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.559687 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.561083 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.562140 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.562914 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.563049 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.563098 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.563186 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":81,"threads_started":1}
I20260928 02:51:16.563926 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.555465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:45839 (local address 127.0.0.1:38486)
0928 02:51:16.556289 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.556302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.556388 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.558099 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.558102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.558120 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:16.558214 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.558221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.559603 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.559607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.560875 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.560886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.563578 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.563582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.563670 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.563693 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.563738 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.563805 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260928 02:51:16.564677 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.555814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.19.139.129:45839)
0928 02:51:16.557738 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.557753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.557774 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:51:16.557842 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.557846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.557978 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.558111 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:16.558342 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.559480 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.562245 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.563317 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.564256 (+ 939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.564400 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.564449 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.564537 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":90,"threads_started":1}
I20260928 02:51:16.564872 27290 tablet_service.cc:1467] Processing CreateTablet for tablet bd030290e03e4ba8b3021b86398da175 (DEFAULT_TABLE table=test-table [id=7067f69129244c31a14f21650c14592f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:16.565866 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd030290e03e4ba8b3021b86398da175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:16.566282 27152 tablet_service.cc:1467] Processing CreateTablet for tablet bd030290e03e4ba8b3021b86398da175 (DEFAULT_TABLE table=test-table [id=7067f69129244c31a14f21650c14592f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:16.567111 27152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd030290e03e4ba8b3021b86398da175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:16.567438 27015 tablet_service.cc:1467] Processing CreateTablet for tablet bd030290e03e4ba8b3021b86398da175 (DEFAULT_TABLE table=test-table [id=7067f69129244c31a14f21650c14592f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:16.568441 27015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd030290e03e4ba8b3021b86398da175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:16.574061 27390 tablet_bootstrap.cc:492] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Bootstrap starting.
I20260928 02:51:16.574836 27391 tablet_bootstrap.cc:492] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Bootstrap starting.
I20260928 02:51:16.576254 27392 tablet_bootstrap.cc:492] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Bootstrap starting.
I20260928 02:51:16.576323 27390 tablet_bootstrap.cc:654] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:16.577135 27391 tablet_bootstrap.cc:654] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:16.577283 27390 log.cc:824] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:16.577975 27391 log.cc:824] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:16.578538 27392 tablet_bootstrap.cc:654] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:16.579324 27392 log.cc:824] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:16.579741 27390 tablet_bootstrap.cc:492] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: No bootstrap required, opened a new log
I20260928 02:51:16.579955 27390 ts_tablet_manager.cc:1397] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 02:51:16.579980 27391 tablet_bootstrap.cc:492] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: No bootstrap required, opened a new log
I20260928 02:51:16.580140 27391 ts_tablet_manager.cc:1397] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:51:16.581498 27392 tablet_bootstrap.cc:492] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: No bootstrap required, opened a new log
I20260928 02:51:16.581664 27392 ts_tablet_manager.cc:1397] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:51:16.583727 27390 raft_consensus.cc:348] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.583976 27390 raft_consensus.cc:374] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:16.584054 27390 raft_consensus.cc:729] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8630532d7243439983b8cff12fa436, State: Initialized, Role: FOLLOWER
I20260928 02:51:16.584599 27390 consensus_queue.cc:260] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.584617 27391 raft_consensus.cc:348] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.584872 27391 raft_consensus.cc:374] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:16.584929 27391 raft_consensus.cc:729] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f75d9c849745b999b55d607315f5a3, State: Initialized, Role: FOLLOWER
I20260928 02:51:16.585415 27391 consensus_queue.cc:260] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.585523 27390 ts_tablet_manager.cc:1428] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 02:51:16.585911 27219 heartbeater.cc:503] Master 127.19.139.190:33543 was elected leader, sending a full tablet report...
I20260928 02:51:16.586181 27392 raft_consensus.cc:348] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.586417 27392 raft_consensus.cc:374] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:16.586462 27392 raft_consensus.cc:729] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9224ae22ca9c41dcbabe9026ec7699aa, State: Initialized, Role: FOLLOWER
I20260928 02:51:16.586889 27392 consensus_queue.cc:260] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.588137 27391 ts_tablet_manager.cc:1428] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260928 02:51:16.588961 27357 heartbeater.cc:503] Master 127.19.139.190:33543 was elected leader, sending a full tablet report...
I20260928 02:51:16.589298 27392 ts_tablet_manager.cc:1428] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:51:16.589650 27082 heartbeater.cc:503] Master 127.19.139.190:33543 was elected leader, sending a full tablet report...
W20260928 02:51:16.612948 27084 tablet.cc:2369] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:51:16.647128 27397 raft_consensus.cc:482] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:16.647352 27397 raft_consensus.cc:504] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.648576 27397 leader_election.cc:288] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839), bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777)
I20260928 02:51:16.652449 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.648906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:41239 (local address 127.19.139.129:45839)
0928 02:51:16.649041 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.649046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.649116 (+ 70us) server_negotiation.cc:407] Connection header received
0928 02:51:16.649251 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.649255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.649308 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.649381 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:16.649667 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.650305 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.651462 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.651853 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.651943 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.651996 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.652253 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.652322 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:51:16.652472 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.648966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:60839 (local address 127.19.139.130:41777)
0928 02:51:16.649086 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.649089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.649349 (+ 260us) server_negotiation.cc:407] Connection header received
0928 02:51:16.649445 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.649448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.649503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.649586 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:16.649915 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.650489 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.651506 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.651925 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.652048 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.652104 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.652260 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.652332 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:16.652501 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.648825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:45839 (local address 127.19.139.131:41239)
0928 02:51:16.649033 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.649048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.649151 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.649412 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.649415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.649441 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:16.649565 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.649572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.650427 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.650431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.651333 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.651343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.651965 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.651969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.652082 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.652107 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.652238 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.652316 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 02:51:16.652503 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.648917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:41777 (local address 127.19.139.131:60839)
0928 02:51:16.649282 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:16.649295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:16.649358 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:16.649598 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:16.649600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:16.649613 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:16.649721 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.649727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.650607 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.650609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.651382 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:16.651390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.652048 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.652052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:16.652127 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.652142 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:51:16.652226 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:51:16.652293 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":1}
I20260928 02:51:16.653158 27172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd030290e03e4ba8b3021b86398da175" candidate_uuid: "38f75d9c849745b999b55d607315f5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8630532d7243439983b8cff12fa436" is_pre_election: true
I20260928 02:51:16.653157 27035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd030290e03e4ba8b3021b86398da175" candidate_uuid: "38f75d9c849745b999b55d607315f5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" is_pre_election: true
I20260928 02:51:16.653479 27172 raft_consensus.cc:2454] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f75d9c849745b999b55d607315f5a3 in term 0.
I20260928 02:51:16.653481 27035 raft_consensus.cc:2454] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f75d9c849745b999b55d607315f5a3 in term 0.
I20260928 02:51:16.654037 27245 leader_election.cc:302] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38f75d9c849745b999b55d607315f5a3, 9224ae22ca9c41dcbabe9026ec7699aa, bc8630532d7243439983b8cff12fa436; no voters:
I20260928 02:51:16.654325 27397 raft_consensus.cc:2781] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:16.654408 27397 raft_consensus.cc:482] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:16.654443 27397 raft_consensus.cc:3037] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:16.655556 27397 raft_consensus.cc:504] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.655912 27397 leader_election.cc:288] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839), bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777)
I20260928 02:51:16.656283 27035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd030290e03e4ba8b3021b86398da175" candidate_uuid: "38f75d9c849745b999b55d607315f5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9224ae22ca9c41dcbabe9026ec7699aa"
I20260928 02:51:16.656405 27035 raft_consensus.cc:3037] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:16.657770 27035 raft_consensus.cc:2454] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f75d9c849745b999b55d607315f5a3 in term 1.
I20260928 02:51:16.658066 27172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd030290e03e4ba8b3021b86398da175" candidate_uuid: "38f75d9c849745b999b55d607315f5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8630532d7243439983b8cff12fa436"
I20260928 02:51:16.658164 27245 leader_election.cc:302] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f75d9c849745b999b55d607315f5a3, 9224ae22ca9c41dcbabe9026ec7699aa; no voters:
I20260928 02:51:16.658214 27172 raft_consensus.cc:3037] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:16.658363 27397 raft_consensus.cc:2781] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:16.658562 27397 raft_consensus.cc:686] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 38f75d9c849745b999b55d607315f5a3, State: Running, Role: LEADER
I20260928 02:51:16.658911 27397 consensus_queue.cc:237] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } }
I20260928 02:51:16.660389 27172 raft_consensus.cc:2454] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f75d9c849745b999b55d607315f5a3 in term 1.
I20260928 02:51:16.662720 26118 catalog_manager.cc:5495] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38f75d9c849745b999b55d607315f5a3 (127.19.139.131). New cstate: current_term: 1 leader_uuid: "38f75d9c849745b999b55d607315f5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } health_report { overall_health: UNKNOWN } } }
I20260928 02:51:16.686852 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.683351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.19.139.131:46441)
0928 02:51:16.683477 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.683480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.683541 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:51:16.683702 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.683705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.683762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.683861 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:16.684068 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.684625 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.685444 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.685646 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.685711 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.686014 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:16.686484 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:16.686575 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.686685 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.686738 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:51:16.693090 27172 raft_consensus.cc:1261] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Refusing update from remote peer 38f75d9c849745b999b55d607315f5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:16.693091 27035 raft_consensus.cc:1261] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 FOLLOWER]: Refusing update from remote peer 38f75d9c849745b999b55d607315f5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:16.693776 27399 consensus_queue.cc:1035] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:16.693966 27397 consensus_queue.cc:1035] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:16.703378 27403 mvcc.cc:205] Tried to move back new op lower bound from 7334149638927568896 to 7334149638797934592. Current Snapshot: MvccSnapshot[applied={T|T < 7334149638927568896}]
I20260928 02:51:16.717314 26116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:16.717531 26116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:16.722565 26116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7064d87b6446440eb11e70ad20437c70: Sending DeleteTablet for 3 replicas of tablet bd030290e03e4ba8b3021b86398da175
I20260928 02:51:16.723361 27015 tablet_service.cc:1514] Processing DeleteTablet for tablet bd030290e03e4ba8b3021b86398da175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:16 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38486
I20260928 02:51:16.723376 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet bd030290e03e4ba8b3021b86398da175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:16 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:50042
I20260928 02:51:16.723474 27152 tablet_service.cc:1514] Processing DeleteTablet for tablet bd030290e03e4ba8b3021b86398da175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:16 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:57790
I20260928 02:51:16.723948 27412 tablet_replica.cc:330] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: stopping tablet replica
I20260928 02:51:16.724201 27413 tablet_replica.cc:330] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: stopping tablet replica
I20260928 02:51:16.724378 27412 raft_consensus.cc:2229] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:16.724579 27413 raft_consensus.cc:2229] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:16.724822 27412 raft_consensus.cc:2258] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:16.724907 27413 raft_consensus.cc:2258] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:16.725371 27414 tablet_replica.cc:330] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: stopping tablet replica
I20260928 02:51:16.725677 27414 raft_consensus.cc:2229] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:16.726152 27414 raft_consensus.cc:2258] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:16.726795 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26086
I20260928 02:51:16.728495 27413 ts_tablet_manager.cc:1905] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:16.728905 27414 ts_tablet_manager.cc:1905] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:16.729848 27412 ts_tablet_manager.cc:1905] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:16.731477 27414 ts_tablet_manager.cc:1918] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:16.731467 27413 ts_tablet_manager.cc:1918] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:16.731573 27414 log.cc:1196] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/bd030290e03e4ba8b3021b86398da175
I20260928 02:51:16.731875 27414 ts_tablet_manager.cc:1939] T bd030290e03e4ba8b3021b86398da175 P bc8630532d7243439983b8cff12fa436: Deleting consensus metadata
I20260928 02:51:16.732232 27413 log.cc:1196] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/bd030290e03e4ba8b3021b86398da175
I20260928 02:51:16.732520 27412 ts_tablet_manager.cc:1918] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:16.732616 27412 log.cc:1196] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/bd030290e03e4ba8b3021b86398da175
I20260928 02:51:16.732906 27412 ts_tablet_manager.cc:1939] T bd030290e03e4ba8b3021b86398da175 P 38f75d9c849745b999b55d607315f5a3: Deleting consensus metadata
I20260928 02:51:16.733975 27413 ts_tablet_manager.cc:1939] T bd030290e03e4ba8b3021b86398da175 P 9224ae22ca9c41dcbabe9026ec7699aa: Deleting consensus metadata
W20260928 02:51:16.735414 27108 connection.cc:535] server connection from 127.0.0.1:57790 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:51:16.735865 26971 connection.cc:441] server connection from 127.0.0.1:38486 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38486 (request call id 1) could send its response
I20260928 02:51:16.743328 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.725250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.19.139.189:33411)
0928 02:51:16.725378 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.725383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.732013 (+ 6630us) server_negotiation.cc:407] Connection header received
0928 02:51:16.732723 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.732727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.732786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.732841 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.733813 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.734565 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.736627 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.736921 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.737036 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.739284 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.739288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.739291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.739320 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.741042 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.741544 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.741549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.741550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.741642 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.741924 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.741927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.741928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.742150 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.742251 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.743015 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.743154 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:51:16.747179 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.737160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.19.139.188:35263)
0928 02:51:16.737338 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.737343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.737997 (+ 654us) server_negotiation.cc:407] Connection header received
0928 02:51:16.738465 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.738469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.738523 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.738577 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.739364 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.740132 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.741545 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.741866 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.741938 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.743489 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.743493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.743496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.743529 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.745239 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.745795 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.745798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.745800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.745887 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.746178 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.746181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.746183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.746395 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.746504 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.746904 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.747016 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:51:16.747184 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.737360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.19.139.189:33411)
0928 02:51:16.737468 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.737472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.738231 (+ 759us) server_negotiation.cc:407] Connection header received
0928 02:51:16.738334 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.738337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.738391 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.738469 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.739330 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.740133 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.741566 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.741936 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.742005 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.743638 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.743642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.743645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.743668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.745255 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.745829 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.745833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.745834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.745923 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.746201 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.746204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.746206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.746401 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.746500 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.746913 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.747029 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:51:16.753998 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:16.725893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.19.139.188:35263)
0928 02:51:16.726004 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:16.726008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:16.745629 (+ 19621us) server_negotiation.cc:407] Connection header received
0928 02:51:16.745797 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:16.745800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:16.745856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:16.745904 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:16.746611 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.747334 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.748747 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:16.748985 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:16.749068 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:16.750367 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:16.750370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:16.750373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:16.750396 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:16.751637 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.752859 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.752863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.752865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.752952 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:16.753201 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:16.753204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:16.753206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:16.753397 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:16.753511 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:16.753725 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:51:16.753845 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:51:17.706254 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:17.705709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:43217)
0928 02:51:17.706036 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:17.706140 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":80,"threads_started":1}
W20260928 02:51:17.706617 27219 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:33543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:17.706815 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:17.706444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:34703)
0928 02:51:17.706633 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:17.706711 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
W20260928 02:51:17.707145 27082 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:33543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:17.709784 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:17.709361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:46193)
0928 02:51:17.709599 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:17.709688 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
W20260928 02:51:17.710028 27357 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:33543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.222708 27426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7064d87b6446440eb11e70ad20437c70)
I20260928 02:51:18.222944 27426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:18.223119 27427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7064d87b6446440eb11e70ad20437c70)
I20260928 02:51:18.223327 27427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:18.224009 27426 leader_election.cc:288] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543), 105043607d974614bb4289128ca69849 (127.19.139.189:33411)
I20260928 02:51:18.224228 27427 leader_election.cc:288] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543), 6f670347ea08426694fe258964841247 (127.19.139.188:35263)
I20260928 02:51:18.224709 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.224248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58046)
0928 02:51:18.224519 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.224606 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
W20260928 02:51:18.224934 26886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.225234 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.224458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58048)
0928 02:51:18.225064 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.225145 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":75,"threads_started":1}
W20260928 02:51:18.225560 26703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.225852 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.225616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58050)
0928 02:51:18.225744 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.225785 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 02:51:18.225984 26886 leader_election.cc:334] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.226406 27431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.226159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58052)
0928 02:51:18.226286 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.226334 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 02:51:18.226504 26703 leader_election.cc:334] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.229174 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.224655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:35263 (local address 127.0.0.1:32946)
0928 02:51:18.225072 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.225085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:18.225162 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:18.225644 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:18.225647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:18.225668 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:18.225778 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:18.225786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.226803 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.226806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:18.228013 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:18.228024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.228845 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.228848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:18.228917 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:18.228938 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:51:18.228999 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:51:18.229062 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":57,"threads_started":1}
I20260928 02:51:18.229223 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.225063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.19.139.188:35263)
0928 02:51:18.225267 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:18.225271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:18.225287 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:51:18.225337 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:18.225340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:18.225424 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:18.225606 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:18.225917 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.226686 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.228149 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.228729 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.228849 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:18.228901 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:18.229027 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:51:18.229103 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20260928 02:51:18.229595 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.224612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33411 (local address 127.0.0.1:56720)
0928 02:51:18.224799 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.224814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:18.224921 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:18.226177 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:18.226180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:18.226201 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:18.226294 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:18.226301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.227141 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.227144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:18.228345 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:18.228354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.229180 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.229183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:18.229273 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:18.229298 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:51:18.229370 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:51:18.229442 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260928 02:51:18.229816 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.224881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.19.139.189:33411)
0928 02:51:18.225382 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:18.225387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:18.225398 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:51:18.225438 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:18.225440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:18.226033 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:18.226167 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:18.226401 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.227030 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.228449 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:18.229060 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:18.229170 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:18.229211 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:18.229441 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:51:18.229554 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":54,"threads_started":1}
I20260928 02:51:18.229871 26911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "105043607d974614bb4289128ca69849" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f670347ea08426694fe258964841247" is_pre_election: true
I20260928 02:51:18.230103 26911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105043607d974614bb4289128ca69849 in term 1.
I20260928 02:51:18.230293 26729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f670347ea08426694fe258964841247" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "105043607d974614bb4289128ca69849" is_pre_election: true
I20260928 02:51:18.230481 26729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f670347ea08426694fe258964841247 in term 1.
I20260928 02:51:18.230520 26706 leader_election.cc:302] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 105043607d974614bb4289128ca69849, 6f670347ea08426694fe258964841247; no voters: 7064d87b6446440eb11e70ad20437c70
I20260928 02:51:18.230829 26886 leader_election.cc:302] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 105043607d974614bb4289128ca69849, 6f670347ea08426694fe258964841247; no voters: 7064d87b6446440eb11e70ad20437c70
I20260928 02:51:18.230895 27427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:51:18.230957 27427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7064d87b6446440eb11e70ad20437c70)
I20260928 02:51:18.230999 27427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:51:18.231016 27426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:51:18.231065 27426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7064d87b6446440eb11e70ad20437c70)
I20260928 02:51:18.231098 27426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:51:18.231956 27427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:18.231997 27426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:18.232331 27427 leader_election.cc:288] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 2 election: Requested vote from peers 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543), 6f670347ea08426694fe258964841247 (127.19.139.188:35263)
I20260928 02:51:18.232348 27426 leader_election.cc:288] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [CANDIDATE]: Term 2 election: Requested vote from peers 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543), 105043607d974614bb4289128ca69849 (127.19.139.189:33411)
I20260928 02:51:18.232642 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.232438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58058)
0928 02:51:18.232533 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.232583 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:51:18.232704 26911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "105043607d974614bb4289128ca69849" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f670347ea08426694fe258964841247"
I20260928 02:51:18.232770 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.232464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58068)
0928 02:51:18.232668 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.232717 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 02:51:18.232851 26911 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 105043607d974614bb4289128ca69849 in current term 2: Already voted for candidate 6f670347ea08426694fe258964841247 in this term.
I20260928 02:51:18.232838 26729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f670347ea08426694fe258964841247" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "105043607d974614bb4289128ca69849"
I20260928 02:51:18.232978 26729 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f670347ea08426694fe258964841247 in current term 2: Already voted for candidate 105043607d974614bb4289128ca69849 in this term.
I20260928 02:51:18.233471 27430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.233143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58072)
0928 02:51:18.233380 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.233426 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260928 02:51:18.233570 26886 leader_election.cc:334] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.233644 26886 leader_election.cc:302] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f670347ea08426694fe258964841247; no voters: 105043607d974614bb4289128ca69849, 7064d87b6446440eb11e70ad20437c70
I20260928 02:51:18.233734 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.233509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58086)
0928 02:51:18.233631 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.233678 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:51:18.233762 27426 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260928 02:51:18.233806 26703 leader_election.cc:334] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:18.233862 26703 leader_election.cc:302] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 105043607d974614bb4289128ca69849; no voters: 6f670347ea08426694fe258964841247, 7064d87b6446440eb11e70ad20437c70
I20260928 02:51:18.233978 27427 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 02:51:18.707824 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.707208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:43149)
0928 02:51:18.707655 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.707730 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":98,"threads_started":1}
I20260928 02:51:18.708041 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.707625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:35977)
0928 02:51:18.707885 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.707946 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":59,"threads_started":1}
I20260928 02:51:18.711119 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:18.710496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:53131)
0928 02:51:18.710950 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:18.711014 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":68,"threads_started":1}
I20260928 02:51:19.709329 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:19.708681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:48353)
0928 02:51:19.709149 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:19.709219 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":112,"threads_started":1}
I20260928 02:51:19.709331 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:19.708843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:44827)
0928 02:51:19.709150 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:19.709221 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":71,"threads_started":1}
I20260928 02:51:19.712304 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:19.711723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:41449)
0928 02:51:19.712112 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:19.712198 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":79,"threads_started":1}
I20260928 02:51:20.021206 27442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:20.021344 27442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:20.021762 27442 leader_election.cc:288] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543), 6f670347ea08426694fe258964841247 (127.19.139.188:35263)
I20260928 02:51:20.022238 26911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "105043607d974614bb4289128ca69849" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f670347ea08426694fe258964841247" is_pre_election: true
I20260928 02:51:20.022360 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.021982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58290)
0928 02:51:20.022207 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.022268 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260928 02:51:20.022395 26911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105043607d974614bb4289128ca69849 in term 2.
I20260928 02:51:20.022755 26706 leader_election.cc:302] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105043607d974614bb4289128ca69849, 6f670347ea08426694fe258964841247; no voters:
I20260928 02:51:20.022962 27442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 02:51:20.023026 27442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:20.023054 27442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:51:20.023253 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.023042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58296)
0928 02:51:20.023149 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.023198 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:51:20.023382 26703 leader_election.cc:334] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:20.024281 27442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:20.024649 27442 leader_election.cc:288] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 3 election: Requested vote from peers 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543), 6f670347ea08426694fe258964841247 (127.19.139.188:35263)
I20260928 02:51:20.024983 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.024745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58304)
0928 02:51:20.024858 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.024912 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:51:20.025070 26911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "105043607d974614bb4289128ca69849" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f670347ea08426694fe258964841247"
I20260928 02:51:20.025221 26911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:51:20.025664 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.025444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58306)
0928 02:51:20.025545 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.025601 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:51:20.025818 26703 leader_election.cc:334] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
I20260928 02:51:20.026465 26911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105043607d974614bb4289128ca69849 in term 3.
I20260928 02:51:20.026749 26706 leader_election.cc:302] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 105043607d974614bb4289128ca69849, 6f670347ea08426694fe258964841247; no voters: 7064d87b6446440eb11e70ad20437c70
I20260928 02:51:20.026901 27442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:51:20.027052 27442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [term 3 LEADER]: Becoming Leader. State: Replica: 105043607d974614bb4289128ca69849, State: Running, Role: LEADER
I20260928 02:51:20.027299 27442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } }
I20260928 02:51:20.028290 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105043607d974614bb4289128ca69849. Latest consensus state: current_term: 3 leader_uuid: "105043607d974614bb4289128ca69849" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:20.028385 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:20.028668 27447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:20.030961 27447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:20.031369 27447 catalog_manager.cc:1259] Loaded cluster ID: 6db09412b7294458b363ab483943b76a
I20260928 02:51:20.031417 27447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:20.031992 27447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:20.032423 27447 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Loaded TSK: 0
I20260928 02:51:20.033084 27447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:20.046528 26718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:20.049762 26718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:20.053001 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.052710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58334)
0928 02:51:20.052841 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.052912 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:51:20.053040 26911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [term 3 FOLLOWER]: Refusing update from remote peer 105043607d974614bb4289128ca69849: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 02:51:20.053501 27442 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 02:51:20.053730 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.053517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.0.0.1:58344)
0928 02:51:20.053615 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.053669 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:51:20.053867 26703 consensus_peers.cc:487] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 -> Peer 7064d87b6446440eb11e70ad20437c70 (127.19.139.190:33543): Couldn't send request to peer 7064d87b6446440eb11e70ad20437c70. Status: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:51:20.054946 27449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105043607d974614bb4289128ca69849. Latest consensus state: current_term: 3 leader_uuid: "105043607d974614bb4289128ca69849" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:20.055065 27449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:20.056213 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "105043607d974614bb4289128ca69849" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:20.056339 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:20.056600 27449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "105043607d974614bb4289128ca69849" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7064d87b6446440eb11e70ad20437c70" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33543 } } peers { permanent_uuid: "105043607d974614bb4289128ca69849" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33411 } attrs { promote: false } } peers { permanent_uuid: "6f670347ea08426694fe258964841247" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 35263 } attrs { promote: false } } }
I20260928 02:51:20.056725 27449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f670347ea08426694fe258964841247 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:51:20.067236 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.062772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:41777 (local address 127.0.0.1:57798)
0928 02:51:20.062897 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.062913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:20.063003 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:20.063628 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:20.063631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:20.063648 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:20.063771 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.063779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.065006 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.065009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.066000 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.066008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.066852 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.066856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.066944 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.066968 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:51:20.067018 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:51:20.067091 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260928 02:51:20.067818 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.062899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.19.139.130:41777)
0928 02:51:20.063367 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:20.063371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:20.063385 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:20.063433 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:20.063443 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:20.063504 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:20.063882 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:20.064008 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.064912 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.066146 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.066728 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.067534 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.067587 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:20.067622 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:51:20.067692 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":91,"threads_started":1}
I20260928 02:51:20.068205 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.063410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.19.139.129:45839)
0928 02:51:20.063621 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:20.063625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:20.064107 (+ 482us) server_negotiation.cc:407] Connection header received
0928 02:51:20.064412 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:20.064415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:20.064468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:20.064543 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:20.064876 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.065448 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.067026 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.067606 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.067714 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.067759 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:20.067923 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:51:20.068040 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20260928 02:51:20.068516 27152 tablet_service.cc:1467] Processing CreateTablet for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 (DEFAULT_TABLE table=test-table [id=5925d59b2eec43a2b5828a5f134d773b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:20.068593 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.063232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:45839 (local address 127.0.0.1:38502)
0928 02:51:20.064018 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.064029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:20.064119 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:20.064653 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:20.064656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:20.064671 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:20.064759 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.064764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.065554 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.065557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.066903 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.066914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.067729 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.067731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.067798 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.067817 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:51:20.067882 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:51:20.068408 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":238,"thread_start_us":79,"threads_started":1}
I20260928 02:51:20.068832 27152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:20.068888 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.063234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46441 (local address 127.0.0.1:50064)
0928 02:51:20.063777 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.063789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:20.064268 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:20.064509 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:20.064512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:20.064529 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:20.064617 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.064622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.065602 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.065605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.066905 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.066914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.068533 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.068537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.068623 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.068644 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:51:20.068714 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:51:20.068777 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":86,"thread_start_us":96,"threads_started":1}
I20260928 02:51:20.069116 27015 tablet_service.cc:1467] Processing CreateTablet for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 (DEFAULT_TABLE table=test-table [id=5925d59b2eec43a2b5828a5f134d773b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:20.069252 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.063357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.19.139.131:46441)
0928 02:51:20.063601 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:20.063606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:20.064208 (+ 602us) server_negotiation.cc:407] Connection header received
0928 02:51:20.064354 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:20.064357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:20.064409 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:20.064499 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:20.064718 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.065516 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.067819 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.068434 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.068553 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.068601 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:20.069040 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 02:51:20.069132 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260928 02:51:20.069375 27015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:20.069979 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 (DEFAULT_TABLE table=test-table [id=5925d59b2eec43a2b5828a5f134d773b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:20.070223 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:20.071293 27460 tablet_bootstrap.cc:492] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa: Bootstrap starting.
I20260928 02:51:20.071903 27461 tablet_bootstrap.cc:492] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Bootstrap starting.
I20260928 02:51:20.072175 27460 tablet_bootstrap.cc:654] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:20.072546 27462 tablet_bootstrap.cc:492] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Bootstrap starting.
I20260928 02:51:20.072760 27461 tablet_bootstrap.cc:654] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:20.073441 27462 tablet_bootstrap.cc:654] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:20.073928 27460 tablet_bootstrap.cc:492] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa: No bootstrap required, opened a new log
I20260928 02:51:20.073928 27461 tablet_bootstrap.cc:492] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: No bootstrap required, opened a new log
I20260928 02:51:20.074021 27461 ts_tablet_manager.cc:1397] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:20.074020 27460 ts_tablet_manager.cc:1397] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:51:20.074445 27462 tablet_bootstrap.cc:492] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: No bootstrap required, opened a new log
I20260928 02:51:20.074522 27462 ts_tablet_manager.cc:1397] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:20.074460 27460 raft_consensus.cc:348] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.074460 27461 raft_consensus.cc:348] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.074568 27460 raft_consensus.cc:374] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:20.074569 27461 raft_consensus.cc:374] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:20.074609 27461 raft_consensus.cc:729] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8630532d7243439983b8cff12fa436, State: Initialized, Role: FOLLOWER
I20260928 02:51:20.074663 27460 raft_consensus.cc:729] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9224ae22ca9c41dcbabe9026ec7699aa, State: Initialized, Role: FOLLOWER
I20260928 02:51:20.074738 27461 consensus_queue.cc:260] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.074877 27460 consensus_queue.cc:260] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.074920 27462 raft_consensus.cc:348] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.075076 27462 raft_consensus.cc:374] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:20.075098 27461 ts_tablet_manager.cc:1428] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:20.075138 27462 raft_consensus.cc:729] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f75d9c849745b999b55d607315f5a3, State: Initialized, Role: FOLLOWER
I20260928 02:51:20.075279 27462 consensus_queue.cc:260] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.075346 27460 ts_tablet_manager.cc:1428] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:20.075649 27462 ts_tablet_manager.cc:1428] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:20.076215 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.075832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:56607)
0928 02:51:20.076052 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.076113 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 02:51:20.077276 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.077011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:49859)
0928 02:51:20.077157 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.077206 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:51:20.077695 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.077127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:40239)
0928 02:51:20.077562 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.077630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
I20260928 02:51:20.078725 27220 heartbeater.cc:503] Master 127.19.139.189:33411 was elected leader, sending a full tablet report...
I20260928 02:51:20.079597 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.079267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:33205)
0928 02:51:20.079478 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.079537 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 02:51:20.079627 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.079407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:42815)
0928 02:51:20.079503 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.079560 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:51:20.079864 27083 heartbeater.cc:503] Master 127.19.139.189:33411 was elected leader, sending a full tablet report...
I20260928 02:51:20.080511 27358 heartbeater.cc:503] Master 127.19.139.189:33411 was elected leader, sending a full tablet report...
I20260928 02:51:20.081029 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.080699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:46209)
0928 02:51:20.080911 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.080961 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260928 02:51:20.259658 27359 tablet.cc:2369] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:51:20.313043 27467 raft_consensus.cc:482] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:20.313225 27467 raft_consensus.cc:504] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
W20260928 02:51:20.314476 27221 tablet.cc:2369] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:51:20.314906 27467 leader_election.cc:288] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777), 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441)
I20260928 02:51:20.319691 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.315149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:45789 (local address 127.19.139.131:46441)
0928 02:51:20.315267 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:20.315272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:20.315288 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:51:20.315345 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:20.315348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:20.315394 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:20.315462 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:20.315762 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.316617 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.318356 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.319010 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.319112 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.319165 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:20.319489 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:51:20.319569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:20.319684 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.314772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46441 (local address 127.19.139.129:45789)
0928 02:51:20.315085 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.315100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:20.315194 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:20.315485 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:20.315487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:20.315500 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:20.315627 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.315634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.316745 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.316749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.318206 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.318218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.319128 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.319133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.319296 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.319345 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:51:20.319462 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:51:20.319517 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":117}
I20260928 02:51:20.319691 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.314782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:41777 (local address 127.19.139.129:52673)
0928 02:51:20.315266 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.315279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:20.315383 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:20.315659 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:20.315662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:20.315678 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:20.315763 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.315767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.316745 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.316749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.318206 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:20.318218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.319140 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.319143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:20.319272 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.319339 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 02:51:20.319417 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:51:20.319503 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":199,"thread_start_us":130,"threads_started":1}
I20260928 02:51:20.319691 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.315332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:52673 (local address 127.19.139.130:41777)
0928 02:51:20.315458 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:20.315462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:20.315477 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:51:20.315520 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:20.315522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:20.315564 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:20.315624 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:20.315867 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.316630 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.318359 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.318980 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.319109 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.319165 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:20.319487 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:51:20.319567 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:51:20.320497 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85aa655cf9411b8b205ff4c8a6e9f6" candidate_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f75d9c849745b999b55d607315f5a3" is_pre_election: true
I20260928 02:51:20.320756 27310 raft_consensus.cc:2454] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9224ae22ca9c41dcbabe9026ec7699aa in term 0.
I20260928 02:51:20.320812 27172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85aa655cf9411b8b205ff4c8a6e9f6" candidate_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8630532d7243439983b8cff12fa436" is_pre_election: true
I20260928 02:51:20.320945 27172 raft_consensus.cc:2454] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9224ae22ca9c41dcbabe9026ec7699aa in term 0.
I20260928 02:51:20.321333 26969 leader_election.cc:302] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38f75d9c849745b999b55d607315f5a3, 9224ae22ca9c41dcbabe9026ec7699aa, bc8630532d7243439983b8cff12fa436; no voters:
I20260928 02:51:20.321853 27467 raft_consensus.cc:2781] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:20.321920 27467 raft_consensus.cc:482] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:20.321986 27467 raft_consensus.cc:3037] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:20.323179 27467 raft_consensus.cc:504] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.323500 27467 leader_election.cc:288] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [CANDIDATE]: Term 1 election: Requested vote from peers bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777), 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441)
I20260928 02:51:20.323830 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85aa655cf9411b8b205ff4c8a6e9f6" candidate_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f75d9c849745b999b55d607315f5a3"
I20260928 02:51:20.323858 27172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85aa655cf9411b8b205ff4c8a6e9f6" candidate_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8630532d7243439983b8cff12fa436"
I20260928 02:51:20.323961 27310 raft_consensus.cc:3037] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:20.323962 27172 raft_consensus.cc:3037] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:20.324870 27172 raft_consensus.cc:2454] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9224ae22ca9c41dcbabe9026ec7699aa in term 1.
I20260928 02:51:20.324975 27310 raft_consensus.cc:2454] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9224ae22ca9c41dcbabe9026ec7699aa in term 1.
I20260928 02:51:20.325150 26970 leader_election.cc:302] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9224ae22ca9c41dcbabe9026ec7699aa, bc8630532d7243439983b8cff12fa436; no voters:
I20260928 02:51:20.325321 27467 raft_consensus.cc:2781] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:20.325541 27467 raft_consensus.cc:686] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [term 1 LEADER]: Becoming Leader. State: Replica: 9224ae22ca9c41dcbabe9026ec7699aa, State: Running, Role: LEADER
I20260928 02:51:20.325928 27467 consensus_queue.cc:237] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } }
I20260928 02:51:20.326249 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.325995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:58279)
0928 02:51:20.326116 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.326175 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:51:20.328583 26718 catalog_manager.cc:5495] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129). New cstate: current_term: 1 leader_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9224ae22ca9c41dcbabe9026ec7699aa" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 45839 } health_report { overall_health: HEALTHY } } }
I20260928 02:51:20.365957 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.362201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.19.139.129:45839)
0928 02:51:20.362327 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:20.362331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:20.362347 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:51:20.362542 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:20.362545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:20.362605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:20.362694 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:20.362951 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.363457 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.364337 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:20.364603 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:20.364693 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:20.365009 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:20.365599 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:20.365700 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:20.365791 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:51:20.365849 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:51:20.368701 27172 raft_consensus.cc:1261] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Refusing update from remote peer 9224ae22ca9c41dcbabe9026ec7699aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:20.368896 27310 raft_consensus.cc:1261] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Refusing update from remote peer 9224ae22ca9c41dcbabe9026ec7699aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:20.369282 27470 consensus_queue.cc:1035] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8630532d7243439983b8cff12fa436" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:20.369544 27467 consensus_queue.cc:1035] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 9224ae22ca9c41dcbabe9026ec7699aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f75d9c849745b999b55d607315f5a3" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:20.371279 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.370972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:57095)
0928 02:51:20.371133 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.371195 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:51:20.372701 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.372445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:50719)
0928 02:51:20.372565 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.372625 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:51:20.375028 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.374743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.129:56983)
0928 02:51:20.374881 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.374936 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:51:20.376255 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.375575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.130:58343)
0928 02:51:20.376091 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.376183 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260928 02:51:20.376559 27475 mvcc.cc:205] Tried to move back new op lower bound from 7334149653985288192 to 7334149653817892864. Current Snapshot: MvccSnapshot[applied={T|T < 7334149653985288192}]
I20260928 02:51:20.377512 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.377180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33543 (local address 127.19.139.131:44699)
0928 02:51:20.377390 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.377443 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:33543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260928 02:51:20.383486 26719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:20.383665 26719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:20.387717 26719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 105043607d974614bb4289128ca69849: Sending DeleteTablet for 3 replicas of tablet 5b85aa655cf9411b8b205ff4c8a6e9f6
I20260928 02:51:20.388499 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:20 UTC) from {username='oryx', principal='oryx/127.19.139.189@KRBTEST.COM'} at 127.0.0.1:50064
I20260928 02:51:20.388893 27482 tablet_replica.cc:330] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: stopping tablet replica
I20260928 02:51:20.388893 27152 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:20 UTC) from {username='oryx', principal='oryx/127.19.139.189@KRBTEST.COM'} at 127.0.0.1:57798
I20260928 02:51:20.389019 27482 raft_consensus.cc:2229] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:20.389011 27015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:20 UTC) from {username='oryx', principal='oryx/127.19.139.189@KRBTEST.COM'} at 127.0.0.1:38502
I20260928 02:51:20.389153 27482 raft_consensus.cc:2258] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:20.389632 27483 tablet_replica.cc:330] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: stopping tablet replica
I20260928 02:51:20.389806 27483 raft_consensus.cc:2229] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:20.389987 27483 raft_consensus.cc:2258] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:20.390034 27482 ts_tablet_manager.cc:1905] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:20.390118 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26863
I20260928 02:51:20.392210 27483 ts_tablet_manager.cc:1905] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:20.392944 27482 ts_tablet_manager.cc:1918] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:20.393056 27482 log.cc:1196] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/5b85aa655cf9411b8b205ff4c8a6e9f6
I20260928 02:51:20.393378 27482 ts_tablet_manager.cc:1939] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P 38f75d9c849745b999b55d607315f5a3: Deleting consensus metadata
I20260928 02:51:20.394210 26704 catalog_manager.cc:4841] TS 38f75d9c849745b999b55d607315f5a3 (127.19.139.131:46441): tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 (table test-table [id=5925d59b2eec43a2b5828a5f134d773b]) successfully deleted
I20260928 02:51:20.394596 27483 ts_tablet_manager.cc:1918] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:20.394659 27483 log.cc:1196] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/5b85aa655cf9411b8b205ff4c8a6e9f6
I20260928 02:51:20.394863 27483 ts_tablet_manager.cc:1939] T 5b85aa655cf9411b8b205ff4c8a6e9f6 P bc8630532d7243439983b8cff12fa436: Deleting consensus metadata
I20260928 02:51:20.395586 26705 catalog_manager.cc:4841] TS bc8630532d7243439983b8cff12fa436 (127.19.139.130:41777): tablet 5b85aa655cf9411b8b205ff4c8a6e9f6 (table test-table [id=5925d59b2eec43a2b5828a5f134d773b]) successfully deleted
I20260928 02:51:20.396644 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 27089
I20260928 02:51:20.396905 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:20.396657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:45839 (local address 127.0.0.1:38526)
0928 02:51:20.396770 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:20.396814 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:51:20.397449 26706 catalog_manager.cc:4584] TS 9224ae22ca9c41dcbabe9026ec7699aa (127.19.139.129:45839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b85aa655cf9411b8b205ff4c8a6e9f6: Network error: Client connection negotiation failed: client connection to 127.19.139.129:45839: connect: Connection refused (error 111)
I20260928 02:51:20.401691 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 27225
I20260928 02:51:20.407125 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26582
I20260928 02:51:20.411778 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 26690
2026-09-28T02:51:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7120 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:20 dist-test-slave-5vc0 krb5kdc[27487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:20 dist-test-slave-5vc0 krb5kdc[27487](info): set up 2 sockets
Sep 28 02:51:20 dist-test-slave-5vc0 krb5kdc[27487](info): commencing operation
krb5kdc: starting...
W20260928 02:51:21.372545 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:21 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab.
Entry for principal oryx/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab.
I20260928 02:51:21.494562 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:33187
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab
--principal=oryx/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:21.605016 27500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:21.605301 27500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:21.605346 27500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:21.609449 27500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:21.609516 27500 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:51:21.609532 27500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:21.609549 27500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:21.609565 27500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:21.609581 27500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:21.615072 27500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:33187
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.190.keytab
--principal=oryx/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:21.616250 27500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:21.617313 27500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:21.625790 27506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:21.625849 27508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:21.625753 27505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:21.626025 27500 server_base.cc:1034] running on GCE node
I20260928 02:51:21.626559 27500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:21.627658 27500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:21.628819 27500 hybrid_clock.cc:648] HybridClock initialized: now 1790563881628804 us; error 32 us; skew 500 ppm
Sep 28 02:51:21 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563881, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:21.631732 27500 init.cc:377] Logged in from keytab as oryx/127.19.139.190@KRBTEST.COM (short username oryx)
I20260928 02:51:21.632795 27500 webserver.cc:460] Webserver started at http://127.19.139.190:36075/ using document root <none> and password file <none>
I20260928 02:51:21.633383 27500 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:21.633450 27500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:21.633644 27500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:21.635766 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "cdd16640b64446c4815a189894881889"
format_stamp: "Formatted at 2026-09-28 02:51:21 on dist-test-slave-5vc0"
server_key: "fbf1400886f345014cac3b9525fa26d1"
server_key_iv: "2655efb36cfcefe0e7284d6c05bb4520"
server_key_version: "encryptionkey@0"
I20260928 02:51:21.636402 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "cdd16640b64446c4815a189894881889"
format_stamp: "Formatted at 2026-09-28 02:51:21 on dist-test-slave-5vc0"
server_key: "fbf1400886f345014cac3b9525fa26d1"
server_key_iv: "2655efb36cfcefe0e7284d6c05bb4520"
server_key_version: "encryptionkey@0"
I20260928 02:51:21.640265 27500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:51:21.642797 27515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:21.644341 27500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:21.644471 27500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "cdd16640b64446c4815a189894881889"
format_stamp: "Formatted at 2026-09-28 02:51:21 on dist-test-slave-5vc0"
server_key: "fbf1400886f345014cac3b9525fa26d1"
server_key_iv: "2655efb36cfcefe0e7284d6c05bb4520"
server_key_version: "encryptionkey@0"
I20260928 02:51:21.644577 27500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:21.661597 27500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:21.664951 27500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:21.665145 27500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:21.673015 27500 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:33187
I20260928 02:51:21.673029 27577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:33187 every 8 connection(s)
I20260928 02:51:21.674134 27500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:21.677003 27578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:21.681078 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 27500
I20260928 02:51:21.681206 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:21.681505 20014 external_mini_cluster.cc:1428] Setting key d1db6a22acd96f2b668611bf0fd00cfb
I20260928 02:51:21.683195 27578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Bootstrap starting.
I20260928 02:51:21.685667 27578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:21.686431 27578 log.cc:824] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:21.688282 27578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: No bootstrap required, opened a new log
Sep 28 02:51:21 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:21.691767 27578 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd16640b64446c4815a189894881889" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33187 } }
I20260928 02:51:21.692018 27578 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:21.692075 27578 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd16640b64446c4815a189894881889, State: Initialized, Role: FOLLOWER
I20260928 02:51:21.692579 27578 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd16640b64446c4815a189894881889" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33187 } }
I20260928 02:51:21.692718 27578 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:21.692785 27578 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:21.692888 27578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:21.694027 27578 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd16640b64446c4815a189894881889" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33187 } }
I20260928 02:51:21.694382 27578 leader_election.cc:302] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd16640b64446c4815a189894881889; no voters:
I20260928 02:51:21.694684 27578 leader_election.cc:288] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:21.694788 27583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:21.695017 27583 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [term 1 LEADER]: Becoming Leader. State: Replica: cdd16640b64446c4815a189894881889, State: Running, Role: LEADER
I20260928 02:51:21.695341 27583 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd16640b64446c4815a189894881889" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33187 } }
I20260928 02:51:21.696007 27578 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:21.696645 27583 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd16640b64446c4815a189894881889. Latest consensus state: current_term: 1 leader_uuid: "cdd16640b64446c4815a189894881889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd16640b64446c4815a189894881889" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33187 } } }
I20260928 02:51:21.696758 27583 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:21.696908 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:21.683039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.19.139.190:33187)
0928 02:51:21.683530 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:21.683548 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:21.683586 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:51:21.684373 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:21.684394 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:21.684757 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:21.685067 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:21.685950 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.687024 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.687914 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.688479 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.688631 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:21.691374 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:21.691380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:21.691394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:21.691421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:21.693265 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:21.693781 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:21.693788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:21.693794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:21.693946 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:21.694252 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:21.694255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:21.694257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:21.694736 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:21.694981 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:21.695295 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:51:21.695838 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":165,"threads_started":1}
I20260928 02:51:21.697402 27584 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd16640b64446c4815a189894881889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd16640b64446c4815a189894881889" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 33187 } } }
I20260928 02:51:21.697500 27584 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:21.699174 27589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:21.701812 27589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:21.707146 27589 catalog_manager.cc:1347] Generated new cluster ID: 74af6bf7872a49959aaa88d079d9ad36
I20260928 02:51:21.707219 27589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:21.717955 27589 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:21.718932 27589 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:21.722860 27589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Generated new TSK 0
I20260928 02:51:21.723610 27589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab.
Entry for principal oryx/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab.
I20260928 02:51:21.761200 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33187
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:21.882897 27604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:21.883184 27604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:21.883234 27604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:21.887336 27604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:21.887413 27604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:21.887456 27604 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:51:21.887492 27604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:21.892915 27604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:21.894019 27604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:21.895066 27604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:21.904847 27612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:21.904847 27610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:21.904912 27604 server_base.cc:1034] running on GCE node
W20260928 02:51:21.904851 27609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:21.905490 27604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:21.906136 27604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:21.907318 27604 hybrid_clock.cc:648] HybridClock initialized: now 1790563881907295 us; error 29 us; skew 500 ppm
Sep 28 02:51:21 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563881, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:21.910137 27604 init.cc:377] Logged in from keytab as oryx/127.19.139.129@KRBTEST.COM (short username oryx)
I20260928 02:51:21.911080 27604 webserver.cc:460] Webserver started at http://127.19.139.129:38829/ using document root <none> and password file <none>
I20260928 02:51:21.911643 27604 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:21.911716 27604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:21.911964 27604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:21.914105 27604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "f9fb438fb3aa44d28cda797cb980a892"
format_stamp: "Formatted at 2026-09-28 02:51:21 on dist-test-slave-5vc0"
server_key: "66391591a576193a8fbf674841962ac7"
server_key_iv: "0a6750fc92c3e83ab30da82c0284a4a0"
server_key_version: "encryptionkey@0"
I20260928 02:51:21.914669 27604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "f9fb438fb3aa44d28cda797cb980a892"
format_stamp: "Formatted at 2026-09-28 02:51:21 on dist-test-slave-5vc0"
server_key: "66391591a576193a8fbf674841962ac7"
server_key_iv: "0a6750fc92c3e83ab30da82c0284a4a0"
server_key_version: "encryptionkey@0"
I20260928 02:51:21.918684 27604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:21.921371 27619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:21.922627 27604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:21.922744 27604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "f9fb438fb3aa44d28cda797cb980a892"
format_stamp: "Formatted at 2026-09-28 02:51:21 on dist-test-slave-5vc0"
server_key: "66391591a576193a8fbf674841962ac7"
server_key_iv: "0a6750fc92c3e83ab30da82c0284a4a0"
server_key_version: "encryptionkey@0"
I20260928 02:51:21.922850 27604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:21.947343 27604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:21.950768 27604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:21.950949 27604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:21.952441 27604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:21.952491 27604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:21.952538 27604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:21.952566 27604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:21.962842 27604 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:38507
Sep 28 02:51:21 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563881, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:21.962898 27734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:38507 every 8 connection(s)
I20260928 02:51:21.963758 27604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:21.966976 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:21.953302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:38113 (local address 127.19.139.190:33187)
0928 02:51:21.953411 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:21.953415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:21.955082 (+ 1667us) server_negotiation.cc:407] Connection header received
0928 02:51:21.956320 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:21.956324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:21.956385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:21.956468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:21.957649 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.958475 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.960229 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.960505 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.960591 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:21.963507 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:21.963511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:21.963514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:21.963544 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:21.965065 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:21.965586 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:21.965589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:21.965591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:21.965679 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:21.965992 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:21.965995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:21.965996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:21.966196 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:21.966270 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:21.966741 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 02:51:21.966853 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:51:21.967730 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:21.953858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.19.139.129:38113)
0928 02:51:21.954497 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:21.954584 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:21.956003 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:21.956651 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:21.956661 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:21.957039 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:21.957514 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:21.957528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.958633 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.958637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:21.960078 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:21.960088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.960664 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.960669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:21.960882 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:21.961578 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:21.961600 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:21.963336 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:21.965168 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:21.965173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:21.965184 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:21.965434 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:21.965771 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:21.965774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:21.965775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:21.965910 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:21.966267 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:21.966271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:21.966525 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 02:51:21.966676 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:51:21.966931 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260928 02:51:21.968636 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 27604
I20260928 02:51:21.968760 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:21.969046 20014 external_mini_cluster.cc:1428] Setting key 4c133fbb8f5c3310a5954d626bbc00ed
I20260928 02:51:21.969121 27735 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33187
I20260928 02:51:21.969259 27735 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:21.969821 27735 heartbeater.cc:511] Master 127.19.139.190:33187 requested a full tablet report, sending...
I20260928 02:51:21.971262 27532 ts_manager.cc:194] Registered new tserver with Master: f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507)
I20260928 02:51:21.972716 27532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:38113
I20260928 02:51:21.978317 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:21.971349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.19.139.129:51967)
0928 02:51:21.971678 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:21.971691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:21.971784 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:21.972006 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:21.972009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:21.972255 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:21.972387 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:21.972395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.973254 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.973257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:21.973998 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:21.974008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.974422 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.974426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:21.974523 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:21.974979 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:21.974993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:21.975380 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:21.976956 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:21.976959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:21.976962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:21.977146 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:21.977394 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:21.977397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:21.977399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:21.977485 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:21.977821 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:21.977824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:21.977910 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:51:21.977993 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:51:21.978133 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260928 02:51:21.978310 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:21.971517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:51967 (local address 127.19.139.190:33187)
0928 02:51:21.971626 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:21.971630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:21.971756 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:51:21.971856 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:21.971859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:21.971900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:21.971981 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:21.972541 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.973127 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.974102 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:21.974325 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:21.974411 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:21.975504 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:21.975507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:21.975510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:21.975534 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:21.976857 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:21.977239 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:21.977242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:21.977244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:21.977316 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:21.977562 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:21.977565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:21.977567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:21.977735 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:21.977819 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:21.978057 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:51:21.978159 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for oryx/127.19.139.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab.
Entry for principal oryx/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab.
I20260928 02:51:21.997717 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33187
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:22.108180 27744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:22.108493 27744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:22.108557 27744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:22.112749 27744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:22.112839 27744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:22.112898 27744 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:51:22.112957 27744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:22.118475 27744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:22.119606 27744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:22.120716 27744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:22.130720 27750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:22.130769 27744 server_base.cc:1034] running on GCE node
W20260928 02:51:22.130728 27749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:22.130739 27752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:22.131484 27744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:22.132146 27744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:22.133361 27744 hybrid_clock.cc:648] HybridClock initialized: now 1790563882133341 us; error 32 us; skew 500 ppm
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563882, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:22.136149 27744 init.cc:377] Logged in from keytab as oryx/127.19.139.130@KRBTEST.COM (short username oryx)
I20260928 02:51:22.137156 27744 webserver.cc:460] Webserver started at http://127.19.139.130:41571/ using document root <none> and password file <none>
I20260928 02:51:22.137722 27744 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:22.137790 27744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:22.138008 27744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:22.140116 27744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "b92b6a1bb04241f3be942a1d6e99131e"
format_stamp: "Formatted at 2026-09-28 02:51:22 on dist-test-slave-5vc0"
server_key: "d02a1774d9e056c3ea73cea11fa5ca0b"
server_key_iv: "f0778eb0584f69e49e2071b77722a557"
server_key_version: "encryptionkey@0"
I20260928 02:51:22.140725 27744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "b92b6a1bb04241f3be942a1d6e99131e"
format_stamp: "Formatted at 2026-09-28 02:51:22 on dist-test-slave-5vc0"
server_key: "d02a1774d9e056c3ea73cea11fa5ca0b"
server_key_iv: "f0778eb0584f69e49e2071b77722a557"
server_key_version: "encryptionkey@0"
I20260928 02:51:22.144780 27744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:22.147409 27759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:22.148732 27744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:22.148829 27744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "b92b6a1bb04241f3be942a1d6e99131e"
format_stamp: "Formatted at 2026-09-28 02:51:22 on dist-test-slave-5vc0"
server_key: "d02a1774d9e056c3ea73cea11fa5ca0b"
server_key_iv: "f0778eb0584f69e49e2071b77722a557"
server_key_version: "encryptionkey@0"
I20260928 02:51:22.148947 27744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:22.169193 27744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:22.172536 27744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:22.172722 27744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:22.174295 27744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:22.174361 27744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:22.174407 27744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:22.174435 27744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563882, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:22.184424 27744 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:35099
I20260928 02:51:22.184451 27874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:35099 every 8 connection(s)
I20260928 02:51:22.185302 27744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:22.187544 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.175167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:60681 (local address 127.19.139.190:33187)
0928 02:51:22.175333 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.175338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.176261 (+ 923us) server_negotiation.cc:407] Connection header received
0928 02:51:22.177370 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.177373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.177434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.177513 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:22.178857 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.179396 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.180771 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.180996 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.181090 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.183767 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:22.183772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:22.183775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:22.183806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:22.185300 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.185909 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.185912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.185914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.186018 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.186455 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.186459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.186460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.186718 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:22.186811 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.187271 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.187379 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:51:22.188349 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.175608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.19.139.130:60681)
0928 02:51:22.176076 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.176129 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.177147 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.177754 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.177764 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.178146 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:22.178703 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.178719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.179576 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.179601 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.180658 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.180670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.181156 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.181160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.181323 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.181922 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:22.181938 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:22.183588 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:22.185416 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.185421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.185436 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.185774 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.186157 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.186160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.186162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.186339 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.186816 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:22.186822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:22.187054 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.187223 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.187519 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260928 02:51:22.189590 27875 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33187
I20260928 02:51:22.189733 27875 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:22.190276 27875 heartbeater.cc:511] Master 127.19.139.190:33187 requested a full tablet report, sending...
I20260928 02:51:22.191412 27531 ts_manager.cc:194] Registered new tserver with Master: b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099)
I20260928 02:51:22.191996 27531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:60681
I20260928 02:51:22.194428 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 27744
I20260928 02:51:22.194543 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:22.194825 20014 external_mini_cluster.cc:1428] Setting key fa003d5ef3ca7ce9c059e48b358fe021
I20260928 02:51:22.201676 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.192449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.19.139.130:60057)
0928 02:51:22.193917 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.193931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.194028 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.194323 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.194326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.194550 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:22.194673 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.194680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.195596 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.195600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.196525 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.196536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.197147 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.197151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.197248 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.197836 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:22.197853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:22.198260 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:22.199974 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.199976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.199978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.200202 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.200550 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.200554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.200556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.200704 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.201194 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:22.201196 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:22.201284 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.201357 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.201521 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":105,"threads_started":1}
I20260928 02:51:22.201720 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.192486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:60057 (local address 127.19.139.190:33187)
0928 02:51:22.192598 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.192602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.194008 (+ 1406us) server_negotiation.cc:407] Connection header received
0928 02:51:22.194110 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.194113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.194176 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.194251 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:22.194789 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.195438 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.196677 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.197012 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.197113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.198408 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:22.198411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:22.198419 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:22.198444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:22.199861 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.200298 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.200301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.200303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.200395 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.200801 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.200805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.200806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.201105 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:22.201202 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.201439 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.201563 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for oryx/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab.
Entry for principal oryx/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab.
I20260928 02:51:22.224296 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:33187
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:22.334308 27884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:22.334594 27884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:22.334661 27884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:22.338816 27884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:22.338886 27884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:22.338941 27884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:51:22.338996 27884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:22.344334 27884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.139.190:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:22.345407 27884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:22.346450 27884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:22.356290 27892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:22.356330 27889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:22.356386 27884 server_base.cc:1034] running on GCE node
W20260928 02:51:22.356289 27890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:22.356971 27884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:22.357611 27884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:22.358808 27884 hybrid_clock.cc:648] HybridClock initialized: now 1790563882358770 us; error 50 us; skew 500 ppm
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563882, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:22.361778 27884 init.cc:377] Logged in from keytab as oryx/127.19.139.131@KRBTEST.COM (short username oryx)
I20260928 02:51:22.362673 27884 webserver.cc:460] Webserver started at http://127.19.139.131:34633/ using document root <none> and password file <none>
I20260928 02:51:22.363216 27884 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:22.363276 27884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:22.363451 27884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:22.365553 27884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "57a3c3b54a934772bd581f2777b4a3f7"
format_stamp: "Formatted at 2026-09-28 02:51:22 on dist-test-slave-5vc0"
server_key: "0a65719c6dd16e04a827b003df7b83b1"
server_key_iv: "cf794a57415c3f56331515d4c630eac4"
server_key_version: "encryptionkey@0"
I20260928 02:51:22.366104 27884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "57a3c3b54a934772bd581f2777b4a3f7"
format_stamp: "Formatted at 2026-09-28 02:51:22 on dist-test-slave-5vc0"
server_key: "0a65719c6dd16e04a827b003df7b83b1"
server_key_iv: "cf794a57415c3f56331515d4c630eac4"
server_key_version: "encryptionkey@0"
I20260928 02:51:22.370364 27884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:22.372951 27899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:22.374303 27884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:22.374430 27884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "57a3c3b54a934772bd581f2777b4a3f7"
format_stamp: "Formatted at 2026-09-28 02:51:22 on dist-test-slave-5vc0"
server_key: "0a65719c6dd16e04a827b003df7b83b1"
server_key_iv: "cf794a57415c3f56331515d4c630eac4"
server_key_version: "encryptionkey@0"
I20260928 02:51:22.374563 27884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:22.399227 27884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:22.402335 27884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:22.402511 27884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:22.404013 27884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:22.404066 27884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:22.404107 27884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:22.404134 27884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563882, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:22.414338 27884 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:35555
I20260928 02:51:22.414418 28014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:35555 every 8 connection(s)
I20260928 02:51:22.415336 27884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:22.417464 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.404933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:39623 (local address 127.19.139.190:33187)
0928 02:51:22.405047 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.405051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.406195 (+ 1144us) server_negotiation.cc:407] Connection header received
0928 02:51:22.407322 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.407326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.407398 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.407496 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:22.408826 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.409440 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.410728 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.410966 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.411059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.413996 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:22.414000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:22.414003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:22.414035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:22.415444 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.415987 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.415992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.415994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.416126 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.416513 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.416516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.416517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.416738 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:22.416816 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.417241 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.417340 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:51:22.418438 27908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.405538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.19.139.131:39623)
0928 02:51:22.406029 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.406081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.407065 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.407689 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.407698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.408124 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:22.408678 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.408692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.409556 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.409559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.410607 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.410618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.411120 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.411124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.411323 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.411969 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:22.411986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:22.413759 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:22.415547 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.415553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.415568 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.415891 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.416247 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.416250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.416252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.416436 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.416816 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:22.416820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:22.417049 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.417201 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.417515 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260928 02:51:22.419756 28015 heartbeater.cc:347] Connected to a master server at 127.19.139.190:33187
I20260928 02:51:22.419919 28015 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:22.420459 28015 heartbeater.cc:511] Master 127.19.139.190:33187 requested a full tablet report, sending...
I20260928 02:51:22.420912 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 27884
I20260928 02:51:22.421044 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:22.421362 20014 external_mini_cluster.cc:1428] Setting key 204f5bb647fb442e820d9a29f551a99b
I20260928 02:51:22.421579 27532 ts_manager.cc:194] Registered new tserver with Master: 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:22.422302 27532 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:39623
I20260928 02:51:22.423427 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:51:22.429246 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.421999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:34911 (local address 127.19.139.190:33187)
0928 02:51:22.422109 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.422114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.422365 (+ 251us) server_negotiation.cc:407] Connection header received
0928 02:51:22.422464 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.422467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.422511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.422598 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:22.423183 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.423745 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.424634 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.424867 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.424961 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.426151 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:22.426155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:22.426157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:22.426180 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:22.427696 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.428075 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.428080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.428082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.428202 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.428431 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.428434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.428436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.428657 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:22.428752 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.429006 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.429115 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:51:22.429258 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.421966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.19.139.131:34911)
0928 02:51:22.422276 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.422290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.422388 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.422708 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.422710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.422902 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:22.423046 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.423054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.423865 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.423869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.424524 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.424534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.424966 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.424970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.425059 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.425569 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:22.425583 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:22.426019 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:22.427788 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.427790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.427792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.427983 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.428280 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:22.428282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:22.428283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:22.428355 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:22.428737 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:22.428739 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:22.428816 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.428896 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.429045 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:51:22.448288 27581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:22.442008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.19.139.190:33187)
0928 02:51:22.442192 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.442196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.442212 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:51:22.442256 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.442258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.442300 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.442390 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:22.443131 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.443699 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.444620 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.444839 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.444931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.448080 (+ 3149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36516: BlockingRecv error: recv got EOF from 127.0.0.1:36516 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 02:51:22 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:22.456775 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.449632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.19.139.190:33187)
0928 02:51:22.449734 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.449737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.449853 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:51:22.449942 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.449945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.449981 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.450049 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:22.450557 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.451041 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.451753 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.451998 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.452073 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.454223 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:22.454226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:22.454228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:22.454252 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:22.455395 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.455783 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.455785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.455786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.455857 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:22.456058 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:22.456060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:22.456061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:22.456287 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:22.456389 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.456563 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.456657 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:51:22.459524 27532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:22.462503 27532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:22.477258 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.472281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.19.139.131:35555)
0928 02:51:22.472710 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.472721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.472741 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:51:22.472811 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.472815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.473014 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.473158 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.473379 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.474440 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.475774 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.476541 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.476679 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.476813 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.477005 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.477113 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":100,"threads_started":1}
I20260928 02:51:22.477250 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.471514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:35555 (local address 127.0.0.1:57740)
0928 02:51:22.471976 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.471989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.472067 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.473149 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.473152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.473168 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.473273 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.473279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.474535 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.474538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.475657 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.475666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.476660 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.476664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.476787 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.476832 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.476957 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.477091 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":182,"thread_start_us":86,"threads_started":1}
I20260928 02:51:22.477497 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.471870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35099 (local address 127.0.0.1:47000)
0928 02:51:22.472092 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.472105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.472209 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.473274 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.473276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.473290 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.473372 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.473378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.474387 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.474390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.475931 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.475937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.477023 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.477027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.477129 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.477152 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.477198 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.477265 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":185,"thread_start_us":82,"threads_started":1}
I20260928 02:51:22.477670 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.471176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38507 (local address 127.0.0.1:38942)
0928 02:51:22.471523 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.471579 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.471711 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.472635 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.472642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.472665 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.472821 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.472829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.474422 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.474425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.475925 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.475936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.477139 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.477143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.477240 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.477264 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.477310 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.477565 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":108,"thread_start_us":82,"threads_started":1}
I20260928 02:51:22.477833 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.472617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.19.139.130:35099)
0928 02:51:22.472901 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.472911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.472928 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:51:22.472987 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.472991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.473137 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.473496 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.473605 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.474279 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.476068 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.476894 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.477382 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.477520 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.477565 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.477705 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
I20260928 02:51:22.478315 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.471281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.19.139.129:38507)
0928 02:51:22.472062 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.472069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.472093 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:51:22.472195 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.472201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.472457 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.472972 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.473109 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.474277 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.476256 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.477019 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.477955 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.478067 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.478105 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.478173 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":125,"threads_started":1}
I20260928 02:51:22.480453 27949 tablet_service.cc:1467] Processing CreateTablet for tablet 0234961983eb4b7197d195bbe7f83298 (DEFAULT_TABLE table=test-table [id=4576a1536c6d4f8c83a63d18fa928ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:22.480495 27809 tablet_service.cc:1467] Processing CreateTablet for tablet 0234961983eb4b7197d195bbe7f83298 (DEFAULT_TABLE table=test-table [id=4576a1536c6d4f8c83a63d18fa928ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:22.480666 27667 tablet_service.cc:1467] Processing CreateTablet for tablet 0234961983eb4b7197d195bbe7f83298 (DEFAULT_TABLE table=test-table [id=4576a1536c6d4f8c83a63d18fa928ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:22.481431 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0234961983eb4b7197d195bbe7f83298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:22.481491 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0234961983eb4b7197d195bbe7f83298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:22.481491 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0234961983eb4b7197d195bbe7f83298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:22.489897 28044 tablet_bootstrap.cc:492] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Bootstrap starting.
I20260928 02:51:22.490294 28043 tablet_bootstrap.cc:492] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Bootstrap starting.
I20260928 02:51:22.492412 28044 tablet_bootstrap.cc:654] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:22.492455 28043 tablet_bootstrap.cc:654] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:22.493295 28043 log.cc:824] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:22.493297 28044 log.cc:824] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:22.494256 28045 tablet_bootstrap.cc:492] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Bootstrap starting.
I20260928 02:51:22.495244 28044 tablet_bootstrap.cc:492] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: No bootstrap required, opened a new log
I20260928 02:51:22.495329 28043 tablet_bootstrap.cc:492] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: No bootstrap required, opened a new log
I20260928 02:51:22.495432 28044 ts_tablet_manager.cc:1397] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:51:22.495493 28043 ts_tablet_manager.cc:1397] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:51:22.496774 28045 tablet_bootstrap.cc:654] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:22.497613 28045 log.cc:824] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:22.499460 28044 raft_consensus.cc:348] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.499462 28043 raft_consensus.cc:348] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.499717 28043 raft_consensus.cc:374] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:22.499717 28044 raft_consensus.cc:374] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:22.499764 28044 raft_consensus.cc:729] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Initialized, Role: FOLLOWER
I20260928 02:51:22.499769 28043 raft_consensus.cc:729] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a3c3b54a934772bd581f2777b4a3f7, State: Initialized, Role: FOLLOWER
I20260928 02:51:22.500206 28044 consensus_queue.cc:260] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.500247 28043 consensus_queue.cc:260] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.501238 28043 ts_tablet_manager.cc:1428] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:51:22.501344 28045 tablet_bootstrap.cc:492] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: No bootstrap required, opened a new log
I20260928 02:51:22.501542 28045 ts_tablet_manager.cc:1397] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:51:22.501717 28015 heartbeater.cc:503] Master 127.19.139.190:33187 was elected leader, sending a full tablet report...
I20260928 02:51:22.503242 28044 ts_tablet_manager.cc:1428] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:51:22.503578 27735 heartbeater.cc:503] Master 127.19.139.190:33187 was elected leader, sending a full tablet report...
I20260928 02:51:22.505451 28045 raft_consensus.cc:348] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.505779 28045 raft_consensus.cc:374] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:22.505842 28045 raft_consensus.cc:729] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Initialized, Role: FOLLOWER
I20260928 02:51:22.506325 28045 consensus_queue.cc:260] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.507200 27875 heartbeater.cc:503] Master 127.19.139.190:33187 was elected leader, sending a full tablet report...
I20260928 02:51:22.507443 28045 ts_tablet_manager.cc:1428] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:51:22.611096 28051 raft_consensus.cc:482] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:22.611337 28051 raft_consensus.cc:504] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.612581 28051 leader_election.cc:288] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:22.617410 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.613015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:42937 (local address 127.19.139.131:35555)
0928 02:51:22.613187 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.613191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.613401 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:51:22.613493 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.613496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.613542 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.613599 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.613832 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.614620 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.616223 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.616855 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.616981 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.617039 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.617206 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.617276 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:51:22.617432 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.612892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:35555 (local address 127.19.139.130:42937)
0928 02:51:22.613309 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.613320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.613403 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.613628 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.613631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.613647 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.613744 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.613749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.614710 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.614714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.616071 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.616083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.616978 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.616980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.617080 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.617108 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.617185 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.617263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":107,"threads_started":1}
I20260928 02:51:22.617441 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.612854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38507 (local address 127.19.139.130:43297)
0928 02:51:22.613044 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.613060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.613156 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.613452 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.613455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.613469 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.613576 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.613582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.614596 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.614598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.616250 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.616265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.617108 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.617111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.617202 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.617219 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.617278 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.617333 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 02:51:22.617483 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.612979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:43297 (local address 127.19.139.129:38507)
0928 02:51:22.613100 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.613104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.613119 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:51:22.613295 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.613298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.613343 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.613421 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.613694 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.614477 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.616367 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.616988 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.617097 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.617131 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.617308 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.617379 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:51:22.618069 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0234961983eb4b7197d195bbe7f83298" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7" is_pre_election: true
I20260928 02:51:22.618069 27689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0234961983eb4b7197d195bbe7f83298" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fb438fb3aa44d28cda797cb980a892" is_pre_election: true
I20260928 02:51:22.618363 27969 raft_consensus.cc:2454] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 0.
I20260928 02:51:22.618366 27689 raft_consensus.cc:2454] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 0.
I20260928 02:51:22.618906 27762 leader_election.cc:302] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57a3c3b54a934772bd581f2777b4a3f7, b92b6a1bb04241f3be942a1d6e99131e, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:22.619176 28051 raft_consensus.cc:2781] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:22.619259 28051 raft_consensus.cc:482] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:22.619309 28051 raft_consensus.cc:3037] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:22.620419 28051 raft_consensus.cc:504] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.620745 28051 leader_election.cc:288] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 election: Requested vote from peers f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:22.621133 27689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0234961983eb4b7197d195bbe7f83298" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fb438fb3aa44d28cda797cb980a892"
I20260928 02:51:22.621136 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0234961983eb4b7197d195bbe7f83298" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7"
I20260928 02:51:22.621254 27969 raft_consensus.cc:3037] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:22.621286 27689 raft_consensus.cc:3037] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:22.622342 27969 raft_consensus.cc:2454] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 1.
I20260928 02:51:22.622344 27689 raft_consensus.cc:2454] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 1.
I20260928 02:51:22.622766 27760 leader_election.cc:302] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57a3c3b54a934772bd581f2777b4a3f7, b92b6a1bb04241f3be942a1d6e99131e, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:22.623008 28051 raft_consensus.cc:2781] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:22.623247 28051 raft_consensus.cc:686] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 1 LEADER]: Becoming Leader. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Running, Role: LEADER
I20260928 02:51:22.623659 28051 consensus_queue.cc:237] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.627893 27531 catalog_manager.cc:5495] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e reported cstate change: term changed from 0 to 1, leader changed from <none> to b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130). New cstate: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } health_report { overall_health: UNKNOWN } } }
I20260928 02:51:22.641571 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.638118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.19.139.130:35099)
0928 02:51:22.638232 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.638236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.638249 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:22.638371 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.638374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.638429 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.638525 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:22.638747 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.639263 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.640181 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.640450 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.640532 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.640792 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:22.641241 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:22.641382 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.641409 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.641463 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:51:22.647426 27689 raft_consensus.cc:1261] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Refusing update from remote peer b92b6a1bb04241f3be942a1d6e99131e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:22.647645 27969 raft_consensus.cc:1261] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Refusing update from remote peer b92b6a1bb04241f3be942a1d6e99131e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:22.648190 28051 consensus_queue.cc:1035] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:22.648391 28053 consensus_queue.cc:1035] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:22.656858 28058 mvcc.cc:205] Tried to move back new op lower bound from 7334149663316422656 to 7334149663230062592. Current Snapshot: MvccSnapshot[applied={T|T < 7334149663316422656}]
I20260928 02:51:22.664670 27531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:22.664855 27531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:22.667191 27531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Sending DeleteTablet for 3 replicas of tablet 0234961983eb4b7197d195bbe7f83298
I20260928 02:51:22.668182 27667 tablet_service.cc:1514] Processing DeleteTablet for tablet 0234961983eb4b7197d195bbe7f83298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:22 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38942
I20260928 02:51:22.668119 27809 tablet_service.cc:1514] Processing DeleteTablet for tablet 0234961983eb4b7197d195bbe7f83298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:22 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:47000
I20260928 02:51:22.668294 27949 tablet_service.cc:1514] Processing DeleteTablet for tablet 0234961983eb4b7197d195bbe7f83298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:22 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:57740
I20260928 02:51:22.668892 28068 tablet_replica.cc:330] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: stopping tablet replica
I20260928 02:51:22.669200 28068 raft_consensus.cc:2229] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:22.669386 27531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:51:22.669528 28068 raft_consensus.cc:2258] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:51:22.669842 27531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:22.670639 28067 tablet_replica.cc:330] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: stopping tablet replica
I20260928 02:51:22.670962 28068 ts_tablet_manager.cc:1905] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:22.670993 28067 raft_consensus.cc:2229] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:22.671300 28067 raft_consensus.cc:2258] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:22.671818 28066 tablet_replica.cc:330] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: stopping tablet replica
I20260928 02:51:22.672181 28066 raft_consensus.cc:2229] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:22.672592 28066 raft_consensus.cc:2258] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:22.672859 28067 ts_tablet_manager.cc:1905] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:22.674336 28068 ts_tablet_manager.cc:1918] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:22.674501 28068 log.cc:1196] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/0234961983eb4b7197d195bbe7f83298
I20260928 02:51:22.674863 28068 ts_tablet_manager.cc:1939] T 0234961983eb4b7197d195bbe7f83298 P 57a3c3b54a934772bd581f2777b4a3f7: Deleting consensus metadata
I20260928 02:51:22.675863 27949 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8e6f3274674298abb0fe03314ad98b (DEFAULT_TABLE table=test-table [id=34999f19a205415790b35e0410cfd5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:22.675873 27667 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8e6f3274674298abb0fe03314ad98b (DEFAULT_TABLE table=test-table [id=34999f19a205415790b35e0410cfd5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:22.675904 27809 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8e6f3274674298abb0fe03314ad98b (DEFAULT_TABLE table=test-table [id=34999f19a205415790b35e0410cfd5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:22.676194 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8e6f3274674298abb0fe03314ad98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:22.676198 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8e6f3274674298abb0fe03314ad98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:22.676205 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8e6f3274674298abb0fe03314ad98b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:22.678351 28044 tablet_bootstrap.cc:492] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Bootstrap starting.
I20260928 02:51:22.678416 28045 tablet_bootstrap.cc:492] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Bootstrap starting.
I20260928 02:51:22.678548 28043 tablet_bootstrap.cc:492] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Bootstrap starting.
I20260928 02:51:22.679327 28045 tablet_bootstrap.cc:654] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:22.679327 28044 tablet_bootstrap.cc:654] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:22.679338 28043 tablet_bootstrap.cc:654] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:22.680460 28043 tablet_bootstrap.cc:492] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: No bootstrap required, opened a new log
I20260928 02:51:22.680547 28043 ts_tablet_manager.cc:1397] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:22.680971 28043 raft_consensus.cc:348] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.681087 28043 raft_consensus.cc:374] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:22.681121 28043 raft_consensus.cc:729] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a3c3b54a934772bd581f2777b4a3f7, State: Initialized, Role: FOLLOWER
I20260928 02:51:22.681231 28043 consensus_queue.cc:260] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.681469 28043 ts_tablet_manager.cc:1428] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:22.681548 27518 catalog_manager.cc:4841] TS 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555): tablet 0234961983eb4b7197d195bbe7f83298 (table test-table [id=4576a1536c6d4f8c83a63d18fa928ae4]) successfully deleted
I20260928 02:51:22.684625 28067 ts_tablet_manager.cc:1918] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:22.684798 28067 log.cc:1196] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/0234961983eb4b7197d195bbe7f83298
I20260928 02:51:22.685034 28044 tablet_bootstrap.cc:492] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: No bootstrap required, opened a new log
I20260928 02:51:22.685124 28044 ts_tablet_manager.cc:1397] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 02:51:22.685262 28067 ts_tablet_manager.cc:1939] T 0234961983eb4b7197d195bbe7f83298 P f9fb438fb3aa44d28cda797cb980a892: Deleting consensus metadata
I20260928 02:51:22.685702 28044 raft_consensus.cc:348] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.685811 28044 raft_consensus.cc:374] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:22.685839 28044 raft_consensus.cc:729] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Initialized, Role: FOLLOWER
I20260928 02:51:22.685958 28044 consensus_queue.cc:260] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.686197 28044 ts_tablet_manager.cc:1428] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:22.686443 27516 catalog_manager.cc:4841] TS f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507): tablet 0234961983eb4b7197d195bbe7f83298 (table test-table [id=4576a1536c6d4f8c83a63d18fa928ae4]) successfully deleted
I20260928 02:51:22.689944 28066 ts_tablet_manager.cc:1905] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:22.693054 28066 ts_tablet_manager.cc:1918] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:22.693171 28066 log.cc:1196] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/0234961983eb4b7197d195bbe7f83298
I20260928 02:51:22.693457 28066 ts_tablet_manager.cc:1939] T 0234961983eb4b7197d195bbe7f83298 P b92b6a1bb04241f3be942a1d6e99131e: Deleting consensus metadata
I20260928 02:51:22.694684 27516 catalog_manager.cc:4841] TS b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099): tablet 0234961983eb4b7197d195bbe7f83298 (table test-table [id=4576a1536c6d4f8c83a63d18fa928ae4]) successfully deleted
I20260928 02:51:22.697173 28045 tablet_bootstrap.cc:492] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: No bootstrap required, opened a new log
I20260928 02:51:22.697283 28045 ts_tablet_manager.cc:1397] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.000s
I20260928 02:51:22.697844 28045 raft_consensus.cc:348] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.697961 28045 raft_consensus.cc:374] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:22.697984 28045 raft_consensus.cc:729] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Initialized, Role: FOLLOWER
I20260928 02:51:22.698128 28045 consensus_queue.cc:260] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.698401 28045 ts_tablet_manager.cc:1428] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260928 02:51:22.715348 27736 tablet.cc:2369] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:51:22.917218 28016 tablet.cc:2369] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:51:22.936659 27876 tablet.cc:2369] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:51:22.944212 28049 raft_consensus.cc:482] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:22.944377 28049 raft_consensus.cc:504] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.945504 28049 leader_election.cc:288] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:22.950930 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.945855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:49609 (local address 127.19.139.130:35099)
0928 02:51:22.946015 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.946020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.946288 (+ 268us) server_negotiation.cc:407] Connection header received
0928 02:51:22.946431 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.946434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.946494 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.946587 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.947006 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.947800 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.949624 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.950236 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.950366 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.950421 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.950722 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.950795 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:51:22.950930 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.945930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:37647 (local address 127.19.139.131:35555)
0928 02:51:22.946071 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:22.946076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:22.946401 (+ 325us) server_negotiation.cc:407] Connection header received
0928 02:51:22.946487 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:22.946490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:22.946532 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:22.946600 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:22.947017 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.947885 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.949624 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.950244 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.950366 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.950421 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:22.950722 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:51:22.950795 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:51:22.951009 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.945831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:35555 (local address 127.19.139.129:37647)
0928 02:51:22.946291 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.946301 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.946393 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.946612 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.946615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.946648 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.946848 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.946855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.947973 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.947977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.949512 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.949519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.950405 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.950408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.950583 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.950624 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.950700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.950819 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":182,"spinlock_wait_cycles":9344,"thread_start_us":108,"threads_started":1}
I20260928 02:51:22.951014 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:22.945785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35099 (local address 127.19.139.129:49609)
0928 02:51:22.946155 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:22.946169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:22.946320 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:22.946612 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:22.946615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:22.946643 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:22.946848 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.946856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.947977 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.947980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.949492 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:22.949504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:22.950393 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:22.950398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:22.950586 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:22.950624 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:51:22.950700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:51:22.950834 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":208,"spinlock_wait_cycles":384,"thread_start_us":124,"threads_started":1}
I20260928 02:51:22.951514 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8e6f3274674298abb0fe03314ad98b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7" is_pre_election: true
I20260928 02:51:22.951521 27829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8e6f3274674298abb0fe03314ad98b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92b6a1bb04241f3be942a1d6e99131e" is_pre_election: true
I20260928 02:51:22.951681 27969 raft_consensus.cc:2454] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 0.
I20260928 02:51:22.951774 27829 raft_consensus.cc:2454] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 0.
I20260928 02:51:22.952045 27622 leader_election.cc:302] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a3c3b54a934772bd581f2777b4a3f7, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:22.952378 28049 raft_consensus.cc:2781] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:22.952445 28049 raft_consensus.cc:482] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:22.952483 28049 raft_consensus.cc:3037] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:22.953563 28049 raft_consensus.cc:504] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.953936 28049 leader_election.cc:288] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 election: Requested vote from peers b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:22.954299 27829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8e6f3274674298abb0fe03314ad98b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92b6a1bb04241f3be942a1d6e99131e"
I20260928 02:51:22.954311 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8e6f3274674298abb0fe03314ad98b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7"
I20260928 02:51:22.954414 27829 raft_consensus.cc:3037] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:22.954428 27969 raft_consensus.cc:3037] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:22.955313 27829 raft_consensus.cc:2454] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 1.
I20260928 02:51:22.955312 27969 raft_consensus.cc:2454] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 1.
I20260928 02:51:22.955586 27620 leader_election.cc:302] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92b6a1bb04241f3be942a1d6e99131e, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:22.955750 28049 raft_consensus.cc:2781] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:22.955947 28049 raft_consensus.cc:686] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 1 LEADER]: Becoming Leader. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Running, Role: LEADER
I20260928 02:51:22.956233 28049 consensus_queue.cc:237] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:22.958494 27531 catalog_manager.cc:5495] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "f9fb438fb3aa44d28cda797cb980a892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563881, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.190@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:23.014235 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.005970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:33187 (local address 127.0.0.1:36524)
0928 02:51:23.006233 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.006246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:23.006406 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:23.006635 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:23.006636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:23.007045 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:23.007154 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.007166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.007908 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.007911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.008529 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.008538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.008974 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.008977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.009045 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.009592 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:23.009605 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:23.011044 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:23.012738 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:23.012747 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:23.012753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:23.012992 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:23.013237 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:23.013240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:23.013242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:23.013314 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:23.013664 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:23.013672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:23.013776 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:51:23.013870 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:51:23.014032 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260928 02:51:23.014232 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.006018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.19.139.190:33187)
0928 02:51:23.006256 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.006260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.006314 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:51:23.006481 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.006484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.006524 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.006602 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:23.007257 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.007808 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.008623 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.008847 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.008923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.011167 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:23.011170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:23.011172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:23.011190 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:23.012635 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:23.013079 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:23.013082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:23.013084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:23.013157 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:23.013396 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:23.013399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:23.013400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:23.013570 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:23.013649 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.013935 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.014096 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260928 02:51:23.018167 27531 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:36524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:51:23.018978 27531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:23.023605 27809 tablet_service.cc:1467] Processing CreateTablet for tablet fb1f316f9211493bb0a731388843df15 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3048ffa571d2409a86f1cdac00edb087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:51:23.023674 27949 tablet_service.cc:1467] Processing CreateTablet for tablet fb1f316f9211493bb0a731388843df15 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3048ffa571d2409a86f1cdac00edb087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:51:23.023617 27667 tablet_service.cc:1467] Processing CreateTablet for tablet fb1f316f9211493bb0a731388843df15 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3048ffa571d2409a86f1cdac00edb087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:51:23.024016 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1f316f9211493bb0a731388843df15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.024016 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1f316f9211493bb0a731388843df15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.024019 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb1f316f9211493bb0a731388843df15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.026328 28045 tablet_bootstrap.cc:492] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: Bootstrap starting.
I20260928 02:51:23.026862 28043 tablet_bootstrap.cc:492] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: Bootstrap starting.
I20260928 02:51:23.027150 28045 tablet_bootstrap.cc:654] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.027724 28043 tablet_bootstrap.cc:654] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.027967 28044 tablet_bootstrap.cc:492] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: Bootstrap starting.
I20260928 02:51:23.028249 28045 tablet_bootstrap.cc:492] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: No bootstrap required, opened a new log
I20260928 02:51:23.028333 28045 ts_tablet_manager.cc:1397] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:23.028759 28044 tablet_bootstrap.cc:654] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.028751 28045 raft_consensus.cc:348] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.028867 28045 raft_consensus.cc:374] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.028874 28043 tablet_bootstrap.cc:492] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: No bootstrap required, opened a new log
I20260928 02:51:23.028905 28045 raft_consensus.cc:729] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.028959 28043 ts_tablet_manager.cc:1397] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:23.029028 28045 consensus_queue.cc:260] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.029300 28045 ts_tablet_manager.cc:1428] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:51:23.029289 28062 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.029384 28043 raft_consensus.cc:348] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.029448 28062 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.029482 28043 raft_consensus.cc:374] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.029512 28043 raft_consensus.cc:729] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a3c3b54a934772bd581f2777b4a3f7, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.029616 28043 consensus_queue.cc:260] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.029844 28043 ts_tablet_manager.cc:1428] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:23.029876 28044 tablet_bootstrap.cc:492] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: No bootstrap required, opened a new log
I20260928 02:51:23.029956 28044 ts_tablet_manager.cc:1397] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:23.030337 28044 raft_consensus.cc:348] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.030455 28044 raft_consensus.cc:374] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.030495 28044 raft_consensus.cc:729] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.030427 28050 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.030560 28050 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.030606 28044 consensus_queue.cc:260] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.030840 28044 ts_tablet_manager.cc:1428] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:23.030999 28049 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.031109 28049 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.102655 28062 raft_consensus.cc:482] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:23.102823 28062 raft_consensus.cc:504] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.103259 28062 leader_election.cc:288] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:23.103838 27689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1f316f9211493bb0a731388843df15" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fb438fb3aa44d28cda797cb980a892" is_pre_election: true
I20260928 02:51:23.103838 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1f316f9211493bb0a731388843df15" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7" is_pre_election: true
I20260928 02:51:23.103994 27689 raft_consensus.cc:2454] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 0.
I20260928 02:51:23.103993 27969 raft_consensus.cc:2454] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 0.
I20260928 02:51:23.104367 27760 leader_election.cc:302] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92b6a1bb04241f3be942a1d6e99131e, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:23.104568 28062 raft_consensus.cc:2781] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:23.104630 28062 raft_consensus.cc:482] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:23.104676 28062 raft_consensus.cc:3037] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.105849 28062 raft_consensus.cc:504] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.106178 28062 leader_election.cc:288] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 election: Requested vote from peers f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:23.106609 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1f316f9211493bb0a731388843df15" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7"
I20260928 02:51:23.106613 27689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb1f316f9211493bb0a731388843df15" candidate_uuid: "b92b6a1bb04241f3be942a1d6e99131e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fb438fb3aa44d28cda797cb980a892"
I20260928 02:51:23.106736 27689 raft_consensus.cc:3037] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.106745 27969 raft_consensus.cc:3037] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.107652 27689 raft_consensus.cc:2454] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 1.
I20260928 02:51:23.107684 27969 raft_consensus.cc:2454] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b92b6a1bb04241f3be942a1d6e99131e in term 1.
I20260928 02:51:23.107937 27760 leader_election.cc:302] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92b6a1bb04241f3be942a1d6e99131e, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:23.108119 28062 raft_consensus.cc:2781] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:23.108215 28062 raft_consensus.cc:686] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [term 1 LEADER]: Becoming Leader. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Running, Role: LEADER
I20260928 02:51:23.108376 28062 consensus_queue.cc:237] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.108975 28053 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: TxnStatusTablet state changed. Reason: New leader b92b6a1bb04241f3be942a1d6e99131e. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.109064 28053 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:51:23.109272 28087 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:51:23.109419 28087 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:51:23.110049 27530 catalog_manager.cc:5495] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e reported cstate change: term changed from 0 to 1, leader changed from <none> to b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130). New cstate: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } health_report { overall_health: UNKNOWN } } }
I20260928 02:51:23.133780 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.130004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35099 (local address 127.0.0.1:47012)
0928 02:51:23.130184 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.130199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:23.130330 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:23.130772 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:23.130776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:23.130783 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:51:23.130884 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.130888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.131875 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.131878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.132571 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.132581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.133134 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.133137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.133211 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.133260 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.133534 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.133543 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:51:23.133590 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:51:23.133662 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:51:23.133809 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.130106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.19.139.130:35099)
0928 02:51:23.130519 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.130536 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.130549 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:23.130603 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.130606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.130655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.130726 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:23.130978 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.131775 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.132677 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.133021 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.133128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.133370 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.133460 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.133525 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.133613 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.133672 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260928 02:51:23.136464 27689 raft_consensus.cc:1261] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Refusing update from remote peer b92b6a1bb04241f3be942a1d6e99131e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.136502 27969 raft_consensus.cc:1261] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Refusing update from remote peer b92b6a1bb04241f3be942a1d6e99131e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.136883 28053 consensus_queue.cc:1035] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.137048 28062 consensus_queue.cc:1035] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.138403 28049 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: TxnStatusTablet state changed. Reason: New leader b92b6a1bb04241f3be942a1d6e99131e. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.138530 28049 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.138536 28050 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: TxnStatusTablet state changed. Reason: New leader b92b6a1bb04241f3be942a1d6e99131e. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.138649 28050 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.140070 28051 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.140120 28053 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.140218 28051 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:51:23.140247 28053 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P b92b6a1bb04241f3be942a1d6e99131e: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:51:23.140332 28056 mvcc.cc:205] Tried to move back new op lower bound from 7334149665322143744 to 7334149665212911616. Current Snapshot: MvccSnapshot[applied={T|T < 7334149665322143744}]
I20260928 02:51:23.141179 28049 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.141285 28049 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P f9fb438fb3aa44d28cda797cb980a892: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.141495 28057 mvcc.cc:205] Tried to move back new op lower bound from 7334149665322143744 to 7334149665212911616. Current Snapshot: MvccSnapshot[applied={T|T < 7334149665322143744}]
I20260928 02:51:23.141674 28050 tablet_replica.cc:439] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b92b6a1bb04241f3be942a1d6e99131e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } } }
I20260928 02:51:23.141772 28050 tablet_replica.cc:442] T fb1f316f9211493bb0a731388843df15 P 57a3c3b54a934772bd581f2777b4a3f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:51:23.151391 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.147054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.19.139.129:38507)
0928 02:51:23.147343 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.147352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.147367 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:51:23.147485 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.147488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.147553 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.147677 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:23.148057 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.148828 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.149837 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.150096 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.150193 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.150503 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.150954 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.151073 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.151145 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.151206 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":126,"threads_started":1}
I20260928 02:51:23.162642 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.159108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:56947 (local address 127.19.139.130:35099)
0928 02:51:23.159229 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.159234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.159447 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:51:23.159527 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.159530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.159580 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.159639 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:23.160193 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.160710 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.161617 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.161830 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.161914 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.162209 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.162286 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.162342 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.162441 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.162497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:51:23.162652 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.159034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35099 (local address 127.19.139.129:56947)
0928 02:51:23.159328 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.159339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:23.159440 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:23.159658 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:23.159660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:23.159667 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:51:23.160055 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.160061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.160814 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.160817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.161506 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.161515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.161951 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.161955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.162037 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.162081 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.162358 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.162364 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:23.162418 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:51:23.162501 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260928 02:51:23.171252 27969 raft_consensus.cc:1261] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Refusing update from remote peer f9fb438fb3aa44d28cda797cb980a892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.171530 27829 raft_consensus.cc:1261] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Refusing update from remote peer f9fb438fb3aa44d28cda797cb980a892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.171778 28049 consensus_queue.cc:1035] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.172423 28049 consensus_queue.cc:1035] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.202133 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.197173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:58195 (local address 127.19.139.129:38507)
0928 02:51:23.197299 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.197304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.197509 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:51:23.197998 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.198002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.198055 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.198096 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:23.198481 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.199264 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.200713 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.201083 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.201191 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.201657 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.201752 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.201792 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.201926 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.201990 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:51:23.202579 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.197069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38507 (local address 127.19.139.130:58195)
0928 02:51:23.197422 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.197435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:23.197724 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:23.198222 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:23.198225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:23.198233 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:51:23.198332 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.198337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.199504 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.199507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.200593 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.200603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.201337 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.201343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.201445 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.201501 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.201845 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.201851 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:23.202340 (+ 489us) client_negotiation.cc:240] Negotiation successful
0928 02:51:23.202440 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20260928 02:51:23.230402 27531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:23.230600 27531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:23.233225 27531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Sending DeleteTablet for 3 replicas of tablet 5f8e6f3274674298abb0fe03314ad98b
I20260928 02:51:23.234058 27667 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8e6f3274674298abb0fe03314ad98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38942
I20260928 02:51:23.234104 27949 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8e6f3274674298abb0fe03314ad98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:57740
I20260928 02:51:23.234731 27809 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8e6f3274674298abb0fe03314ad98b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:47000
I20260928 02:51:23.238229 28106 tablet_replica.cc:330] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: stopping tablet replica
I20260928 02:51:23.238469 28108 tablet_replica.cc:330] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: stopping tablet replica
I20260928 02:51:23.238652 28108 raft_consensus.cc:2229] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:23.238834 28106 raft_consensus.cc:2229] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:23.239033 28108 raft_consensus.cc:2258] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.239387 28106 raft_consensus.cc:2258] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.240262 28108 ts_tablet_manager.cc:1905] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.240974 28106 ts_tablet_manager.cc:1905] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.247010 28106 ts_tablet_manager.cc:1918] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:51:23.247323 28107 tablet_replica.cc:330] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: stopping tablet replica
I20260928 02:51:23.247494 28107 raft_consensus.cc:2229] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:23.247740 28107 raft_consensus.cc:2258] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.248555 28107 ts_tablet_manager.cc:1905] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[27487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 02:51:23.251652 28107 ts_tablet_manager.cc:1918] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:51:23.251839 28106 log.cc:1196] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/5f8e6f3274674298abb0fe03314ad98b
I20260928 02:51:23.244103 28108 ts_tablet_manager.cc:1918] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:51:23.252357 28108 log.cc:1196] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/5f8e6f3274674298abb0fe03314ad98b
I20260928 02:51:23.252673 28108 ts_tablet_manager.cc:1939] T 5f8e6f3274674298abb0fe03314ad98b P b92b6a1bb04241f3be942a1d6e99131e: Deleting consensus metadata
I20260928 02:51:23.252827 28106 ts_tablet_manager.cc:1939] T 5f8e6f3274674298abb0fe03314ad98b P f9fb438fb3aa44d28cda797cb980a892: Deleting consensus metadata
I20260928 02:51:23.253988 27516 catalog_manager.cc:4841] TS f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507): tablet 5f8e6f3274674298abb0fe03314ad98b (table test-table [id=34999f19a205415790b35e0410cfd5d6]) successfully deleted
I20260928 02:51:23.254422 27516 catalog_manager.cc:4841] TS b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099): tablet 5f8e6f3274674298abb0fe03314ad98b (table test-table [id=34999f19a205415790b35e0410cfd5d6]) successfully deleted
I20260928 02:51:23.254673 28107 log.cc:1196] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/5f8e6f3274674298abb0fe03314ad98b
I20260928 02:51:23.254998 28107 ts_tablet_manager.cc:1939] T 5f8e6f3274674298abb0fe03314ad98b P 57a3c3b54a934772bd581f2777b4a3f7: Deleting consensus metadata
I20260928 02:51:23.255699 27518 catalog_manager.cc:4841] TS 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555): tablet 5f8e6f3274674298abb0fe03314ad98b (table test-table [id=34999f19a205415790b35e0410cfd5d6]) successfully deleted
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:51:23.268128 28082 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:23.259907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.19.139.190:33187)
0928 02:51:23.260039 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.260044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.260393 (+ 349us) server_negotiation.cc:407] Connection header received
0928 02:51:23.260533 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.260536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.260597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.260687 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:23.261578 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.262410 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.263671 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.264047 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.264180 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.267985 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36526: BlockingRecv error: recv got EOF from 127.0.0.1:36526 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[27487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM
I20260928 02:51:23.279556 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.269818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.19.139.190:33187)
0928 02:51:23.269960 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.269964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.270386 (+ 422us) server_negotiation.cc:407] Connection header received
0928 02:51:23.270495 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.270499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.270552 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.270631 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:23.271326 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.272132 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.273220 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.273577 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.273676 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.276509 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:23.276513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:23.276516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:23.276552 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:23.277966 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:23.278414 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:23.278417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:23.278418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:23.278494 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:23.278761 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:23.278763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:23.278764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:23.278966 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:23.279081 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.279290 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.279397 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:51:23.282259 27531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:23.282729 27531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:23.287556 27667 tablet_service.cc:1467] Processing CreateTablet for tablet 07d81827bd894811a33a634514494062 (DEFAULT_TABLE table=test-table [id=9a64db050aa347ee8632acfb739978f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:23.287559 27809 tablet_service.cc:1467] Processing CreateTablet for tablet 07d81827bd894811a33a634514494062 (DEFAULT_TABLE table=test-table [id=9a64db050aa347ee8632acfb739978f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:23.287659 27949 tablet_service.cc:1467] Processing CreateTablet for tablet 07d81827bd894811a33a634514494062 (DEFAULT_TABLE table=test-table [id=9a64db050aa347ee8632acfb739978f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:23.287855 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d81827bd894811a33a634514494062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.287856 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d81827bd894811a33a634514494062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.287885 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d81827bd894811a33a634514494062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.290189 28045 tablet_bootstrap.cc:492] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Bootstrap starting.
I20260928 02:51:23.290290 28044 tablet_bootstrap.cc:492] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Bootstrap starting.
I20260928 02:51:23.290915 28045 tablet_bootstrap.cc:654] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.291061 28044 tablet_bootstrap.cc:654] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.291630 28043 tablet_bootstrap.cc:492] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Bootstrap starting.
I20260928 02:51:23.291918 28045 tablet_bootstrap.cc:492] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: No bootstrap required, opened a new log
I20260928 02:51:23.291997 28045 ts_tablet_manager.cc:1397] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:23.292431 28045 raft_consensus.cc:348] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.292550 28043 tablet_bootstrap.cc:654] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.292558 28045 raft_consensus.cc:374] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.292589 28045 raft_consensus.cc:729] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.292730 28045 consensus_queue.cc:260] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.292999 28045 ts_tablet_manager.cc:1428] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:23.294307 28043 tablet_bootstrap.cc:492] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: No bootstrap required, opened a new log
I20260928 02:51:23.294392 28043 ts_tablet_manager.cc:1397] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:51:23.294843 28043 raft_consensus.cc:348] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.294960 28043 raft_consensus.cc:374] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.294991 28043 raft_consensus.cc:729] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a3c3b54a934772bd581f2777b4a3f7, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.295109 28043 consensus_queue.cc:260] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.295343 28044 tablet_bootstrap.cc:492] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: No bootstrap required, opened a new log
I20260928 02:51:23.295387 28043 ts_tablet_manager.cc:1428] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:51:23.295428 28044 ts_tablet_manager.cc:1397] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 02:51:23.295912 28044 raft_consensus.cc:348] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.296053 28044 raft_consensus.cc:374] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.296088 28044 raft_consensus.cc:729] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.296237 28044 consensus_queue.cc:260] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.296510 28044 ts_tablet_manager.cc:1428] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:51:23.428277 28050 raft_consensus.cc:482] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:23.428437 28050 raft_consensus.cc:504] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.429913 28050 leader_election.cc:288] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507), b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099)
I20260928 02:51:23.433980 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.430589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35099 (local address 127.19.139.131:44089)
0928 02:51:23.430914 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.430927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:23.431026 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:23.431232 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:23.431235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:23.431249 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:23.431336 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.431342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.432060 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.432063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.432982 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.432991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.433575 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.433579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.433662 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.433684 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:51:23.433747 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:51:23.433816 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":61,"threads_started":1}
I20260928 02:51:23.433964 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.430700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:44089 (local address 127.19.139.130:35099)
0928 02:51:23.430818 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.430822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.431007 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:51:23.431110 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.431114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.431153 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.431214 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:23.431468 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.431957 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.433092 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.433475 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.433575 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.433628 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.433767 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.433853 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:23.434417 27829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d81827bd894811a33a634514494062" candidate_uuid: "57a3c3b54a934772bd581f2777b4a3f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92b6a1bb04241f3be942a1d6e99131e" is_pre_election: true
I20260928 02:51:23.434576 27829 raft_consensus.cc:2454] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57a3c3b54a934772bd581f2777b4a3f7 in term 0.
I20260928 02:51:23.434844 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.430212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38507 (local address 127.19.139.131:54573)
0928 02:51:23.430499 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.430514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:23.430687 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:23.431074 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:23.431076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:23.431092 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:23.431189 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.431194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.432245 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.432249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.433497 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:23.433505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.434480 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.434483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:23.434567 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.434585 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:51:23.434645 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:51:23.434705 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260928 02:51:23.434907 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.430304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:54573 (local address 127.19.139.129:38507)
0928 02:51:23.430453 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.430458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.430607 (+ 149us) server_negotiation.cc:407] Connection header received
0928 02:51:23.430842 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.430846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.430896 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.430943 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:23.431289 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.432088 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.433628 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.434263 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.434349 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.434396 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.434695 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.434784 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:51:23.434964 27900 leader_election.cc:302] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a3c3b54a934772bd581f2777b4a3f7, b92b6a1bb04241f3be942a1d6e99131e; no voters:
I20260928 02:51:23.435235 28050 raft_consensus.cc:2781] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:23.435297 28050 raft_consensus.cc:482] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:23.435353 28050 raft_consensus.cc:3037] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.435531 27689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d81827bd894811a33a634514494062" candidate_uuid: "57a3c3b54a934772bd581f2777b4a3f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fb438fb3aa44d28cda797cb980a892" is_pre_election: true
I20260928 02:51:23.435654 27689 raft_consensus.cc:2454] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57a3c3b54a934772bd581f2777b4a3f7 in term 0.
I20260928 02:51:23.436535 28050 raft_consensus.cc:504] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.436865 28050 leader_election.cc:288] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [CANDIDATE]: Term 1 election: Requested vote from peers f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507), b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099)
I20260928 02:51:23.437175 27689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d81827bd894811a33a634514494062" candidate_uuid: "57a3c3b54a934772bd581f2777b4a3f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fb438fb3aa44d28cda797cb980a892"
I20260928 02:51:23.437292 27689 raft_consensus.cc:3037] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.437361 27829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d81827bd894811a33a634514494062" candidate_uuid: "57a3c3b54a934772bd581f2777b4a3f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92b6a1bb04241f3be942a1d6e99131e"
I20260928 02:51:23.437489 27829 raft_consensus.cc:3037] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.438510 27689 raft_consensus.cc:2454] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57a3c3b54a934772bd581f2777b4a3f7 in term 1.
I20260928 02:51:23.438587 27829 raft_consensus.cc:2454] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57a3c3b54a934772bd581f2777b4a3f7 in term 1.
I20260928 02:51:23.438831 27900 leader_election.cc:302] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a3c3b54a934772bd581f2777b4a3f7, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:23.439000 28050 raft_consensus.cc:2781] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:23.439204 28050 raft_consensus.cc:686] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 LEADER]: Becoming Leader. State: Replica: 57a3c3b54a934772bd581f2777b4a3f7, State: Running, Role: LEADER
I20260928 02:51:23.439445 28050 consensus_queue.cc:237] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.441602 27530 catalog_manager.cc:5495] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131). New cstate: current_term: 1 leader_uuid: "57a3c3b54a934772bd581f2777b4a3f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } health_report { overall_health: HEALTHY } } }
I20260928 02:51:23.458000 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.454491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.19.139.131:35555)
0928 02:51:23.454721 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.454724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.454737 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:23.454812 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.454814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.454867 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.454971 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:23.455149 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.455678 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.456572 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.456807 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.456898 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.457143 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.457614 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.457706 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.457809 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.457865 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260928 02:51:23.460698 27829 raft_consensus.cc:1261] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Refusing update from remote peer 57a3c3b54a934772bd581f2777b4a3f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.460749 27689 raft_consensus.cc:1261] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Refusing update from remote peer 57a3c3b54a934772bd581f2777b4a3f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.461223 28050 consensus_queue.cc:1035] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.461455 28126 consensus_queue.cc:1035] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.471994 27530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:23.472141 27530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:23.474205 27530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Sending DeleteTablet for 3 replicas of tablet 07d81827bd894811a33a634514494062
I20260928 02:51:23.474792 27809 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d81827bd894811a33a634514494062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:47000
I20260928 02:51:23.474992 28108 tablet_replica.cc:330] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: stopping tablet replica
I20260928 02:51:23.475114 28108 raft_consensus.cc:2229] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:23.475132 27949 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d81827bd894811a33a634514494062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:57740
I20260928 02:51:23.475251 28108 raft_consensus.cc:2258] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.475298 28107 tablet_replica.cc:330] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: stopping tablet replica
I20260928 02:51:23.475407 28107 raft_consensus.cc:2229] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:23.475684 28107 raft_consensus.cc:2258] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.475703 27667 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d81827bd894811a33a634514494062 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38942
I20260928 02:51:23.476244 28108 ts_tablet_manager.cc:1905] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.476536 28107 ts_tablet_manager.cc:1905] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.478685 28107 ts_tablet_manager.cc:1918] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:23.478760 28107 log.cc:1196] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/07d81827bd894811a33a634514494062
I20260928 02:51:23.478848 28108 ts_tablet_manager.cc:1918] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:23.478925 28108 log.cc:1196] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/07d81827bd894811a33a634514494062
I20260928 02:51:23.479003 28107 ts_tablet_manager.cc:1939] T 07d81827bd894811a33a634514494062 P 57a3c3b54a934772bd581f2777b4a3f7: Deleting consensus metadata
I20260928 02:51:23.479187 28108 ts_tablet_manager.cc:1939] T 07d81827bd894811a33a634514494062 P b92b6a1bb04241f3be942a1d6e99131e: Deleting consensus metadata
I20260928 02:51:23.479697 27518 catalog_manager.cc:4841] TS 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555): tablet 07d81827bd894811a33a634514494062 (table test-table [id=9a64db050aa347ee8632acfb739978f4]) successfully deleted
I20260928 02:51:23.479736 27531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:51:23.479863 28106 tablet_replica.cc:330] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: stopping tablet replica
I20260928 02:51:23.480000 28106 raft_consensus.cc:2229] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:23.480139 28106 raft_consensus.cc:2258] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:51:23.480221 27531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:23.480360 27516 catalog_manager.cc:4841] TS b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099): tablet 07d81827bd894811a33a634514494062 (table test-table [id=9a64db050aa347ee8632acfb739978f4]) successfully deleted
I20260928 02:51:23.481009 28106 ts_tablet_manager.cc:1905] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.483781 28106 ts_tablet_manager.cc:1918] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:23.483875 28106 log.cc:1196] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/07d81827bd894811a33a634514494062
I20260928 02:51:23.484117 28106 ts_tablet_manager.cc:1939] T 07d81827bd894811a33a634514494062 P f9fb438fb3aa44d28cda797cb980a892: Deleting consensus metadata
I20260928 02:51:23.484912 27516 catalog_manager.cc:4841] TS f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507): tablet 07d81827bd894811a33a634514494062 (table test-table [id=9a64db050aa347ee8632acfb739978f4]) successfully deleted
I20260928 02:51:23.487175 27667 tablet_service.cc:1467] Processing CreateTablet for tablet d0970f6e7bf743609fac98bb3d22e34b (DEFAULT_TABLE table=test-table [id=8bb4758d7ac34744858b06fe3f81ea09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:23.487228 27809 tablet_service.cc:1467] Processing CreateTablet for tablet d0970f6e7bf743609fac98bb3d22e34b (DEFAULT_TABLE table=test-table [id=8bb4758d7ac34744858b06fe3f81ea09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:23.487349 27949 tablet_service.cc:1467] Processing CreateTablet for tablet d0970f6e7bf743609fac98bb3d22e34b (DEFAULT_TABLE table=test-table [id=8bb4758d7ac34744858b06fe3f81ea09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:23.487460 27667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0970f6e7bf743609fac98bb3d22e34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.487483 27809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0970f6e7bf743609fac98bb3d22e34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.487584 27949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0970f6e7bf743609fac98bb3d22e34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:23.489852 28045 tablet_bootstrap.cc:492] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Bootstrap starting.
I20260928 02:51:23.489903 28044 tablet_bootstrap.cc:492] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892: Bootstrap starting.
I20260928 02:51:23.490569 28044 tablet_bootstrap.cc:654] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.490640 28045 tablet_bootstrap.cc:654] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.491065 28043 tablet_bootstrap.cc:492] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Bootstrap starting.
I20260928 02:51:23.491693 28044 tablet_bootstrap.cc:492] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892: No bootstrap required, opened a new log
I20260928 02:51:23.491765 28044 ts_tablet_manager.cc:1397] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:51:23.491792 28043 tablet_bootstrap.cc:654] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:23.492185 28044 raft_consensus.cc:348] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.492293 28044 raft_consensus.cc:374] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.492322 28044 raft_consensus.cc:729] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.492439 28044 consensus_queue.cc:260] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.492710 28044 ts_tablet_manager.cc:1428] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:51:23.492890 28043 tablet_bootstrap.cc:492] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: No bootstrap required, opened a new log
I20260928 02:51:23.492954 28043 ts_tablet_manager.cc:1397] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:51:23.493276 28043 raft_consensus.cc:348] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.493371 28043 raft_consensus.cc:374] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.493402 28043 raft_consensus.cc:729] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a3c3b54a934772bd581f2777b4a3f7, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.493512 28043 consensus_queue.cc:260] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.493738 28043 ts_tablet_manager.cc:1428] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:51:23.495713 28045 tablet_bootstrap.cc:492] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: No bootstrap required, opened a new log
I20260928 02:51:23.495810 28045 ts_tablet_manager.cc:1397] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 02:51:23.496281 28045 raft_consensus.cc:348] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.496413 28045 raft_consensus.cc:374] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:23.496444 28045 raft_consensus.cc:729] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92b6a1bb04241f3be942a1d6e99131e, State: Initialized, Role: FOLLOWER
I20260928 02:51:23.496563 28045 consensus_queue.cc:260] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.496822 28045 ts_tablet_manager.cc:1428] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:51:23.596580 28074 raft_consensus.cc:482] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:23.596717 28074 raft_consensus.cc:504] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.597186 28074 leader_election.cc:288] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:23.597679 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0970f6e7bf743609fac98bb3d22e34b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7" is_pre_election: true
I20260928 02:51:23.597697 27829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0970f6e7bf743609fac98bb3d22e34b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92b6a1bb04241f3be942a1d6e99131e" is_pre_election: true
I20260928 02:51:23.597833 27829 raft_consensus.cc:2454] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 0.
I20260928 02:51:23.597833 27969 raft_consensus.cc:2454] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 0.
I20260928 02:51:23.598129 27620 leader_election.cc:302] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b92b6a1bb04241f3be942a1d6e99131e, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:23.598299 28074 raft_consensus.cc:2781] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:23.598356 28074 raft_consensus.cc:482] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:23.598405 28074 raft_consensus.cc:3037] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.599406 28074 raft_consensus.cc:504] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.599758 28074 leader_election.cc:288] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 election: Requested vote from peers b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099), 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555)
I20260928 02:51:23.600095 27969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0970f6e7bf743609fac98bb3d22e34b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a3c3b54a934772bd581f2777b4a3f7"
I20260928 02:51:23.600116 27829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0970f6e7bf743609fac98bb3d22e34b" candidate_uuid: "f9fb438fb3aa44d28cda797cb980a892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92b6a1bb04241f3be942a1d6e99131e"
I20260928 02:51:23.600270 27969 raft_consensus.cc:3037] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.600306 27829 raft_consensus.cc:3037] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:23.601220 27969 raft_consensus.cc:2454] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 1.
I20260928 02:51:23.601435 27829 raft_consensus.cc:2454] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fb438fb3aa44d28cda797cb980a892 in term 1.
I20260928 02:51:23.601521 27622 leader_election.cc:302] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a3c3b54a934772bd581f2777b4a3f7, f9fb438fb3aa44d28cda797cb980a892; no voters:
I20260928 02:51:23.601693 28074 raft_consensus.cc:2781] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:23.601788 28074 raft_consensus.cc:686] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [term 1 LEADER]: Becoming Leader. State: Replica: f9fb438fb3aa44d28cda797cb980a892, State: Running, Role: LEADER
I20260928 02:51:23.601941 28074 consensus_queue.cc:237] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } }
I20260928 02:51:23.603592 27531 catalog_manager.cc:5495] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "f9fb438fb3aa44d28cda797cb980a892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fb438fb3aa44d28cda797cb980a892" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 } health_report { overall_health: UNKNOWN } } }
I20260928 02:51:23.629964 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.626964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.19.139.129:38507)
0928 02:51:23.627088 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:23.627093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:23.627107 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:23.627232 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:23.627236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:23.627284 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:23.627344 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:23.627654 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.628209 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.629016 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:23.629239 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:23.629322 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:23.629544 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:23.629619 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:23.629674 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:23.629775 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:51:23.629829 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:51:23.635456 27969 raft_consensus.cc:1261] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Refusing update from remote peer f9fb438fb3aa44d28cda797cb980a892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.635843 27829 raft_consensus.cc:1261] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Refusing update from remote peer f9fb438fb3aa44d28cda797cb980a892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:23.635870 28074 consensus_queue.cc:1035] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a3c3b54a934772bd581f2777b4a3f7" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.636174 28074 consensus_queue.cc:1035] T d0970f6e7bf743609fac98bb3d22e34b P f9fb438fb3aa44d28cda797cb980a892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92b6a1bb04241f3be942a1d6e99131e" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:23.661218 27531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:23.661365 27531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:23.663126 27531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd16640b64446c4815a189894881889: Sending DeleteTablet for 3 replicas of tablet d0970f6e7bf743609fac98bb3d22e34b
I20260928 02:51:23.663709 27949 tablet_service.cc:1514] Processing DeleteTablet for tablet d0970f6e7bf743609fac98bb3d22e34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:57740
I20260928 02:51:23.663895 28107 tablet_replica.cc:330] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: stopping tablet replica
I20260928 02:51:23.664016 28107 raft_consensus.cc:2229] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:23.664124 28107 raft_consensus.cc:2258] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.664376 27667 tablet_service.cc:1514] Processing DeleteTablet for tablet d0970f6e7bf743609fac98bb3d22e34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38942
I20260928 02:51:23.664516 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 27604
I20260928 02:51:23.666152 27809 tablet_service.cc:1514] Processing DeleteTablet for tablet d0970f6e7bf743609fac98bb3d22e34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:23 UTC) from {username='oryx', principal='oryx/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:47000
I20260928 02:51:23.666193 28107 ts_tablet_manager.cc:1905] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.666337 28108 tablet_replica.cc:330] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: stopping tablet replica
I20260928 02:51:23.666471 28108 raft_consensus.cc:2229] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:23.666581 28108 raft_consensus.cc:2258] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:23.667173 28108 ts_tablet_manager.cc:1905] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:23.668901 28107 ts_tablet_manager.cc:1918] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:51:23.669010 28107 log.cc:1196] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/d0970f6e7bf743609fac98bb3d22e34b
I20260928 02:51:23.669261 28107 ts_tablet_manager.cc:1939] T d0970f6e7bf743609fac98bb3d22e34b P 57a3c3b54a934772bd581f2777b4a3f7: Deleting consensus metadata
I20260928 02:51:23.669924 28108 ts_tablet_manager.cc:1918] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:51:23.670009 28108 log.cc:1196] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/d0970f6e7bf743609fac98bb3d22e34b
I20260928 02:51:23.670040 27518 catalog_manager.cc:4841] TS 57a3c3b54a934772bd581f2777b4a3f7 (127.19.139.131:35555): tablet d0970f6e7bf743609fac98bb3d22e34b (table test-table [id=8bb4758d7ac34744858b06fe3f81ea09]) successfully deleted
I20260928 02:51:23.670229 28108 ts_tablet_manager.cc:1939] T d0970f6e7bf743609fac98bb3d22e34b P b92b6a1bb04241f3be942a1d6e99131e: Deleting consensus metadata
I20260928 02:51:23.670915 27516 catalog_manager.cc:4841] TS b92b6a1bb04241f3be942a1d6e99131e (127.19.139.130:35099): tablet d0970f6e7bf743609fac98bb3d22e34b (table test-table [id=8bb4758d7ac34744858b06fe3f81ea09]) successfully deleted
W20260928 02:51:23.671358 27516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:38507 (error 108)
I20260928 02:51:23.672077 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 27744
I20260928 02:51:23.672390 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:23.672017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:38507 (local address 127.0.0.1:38974)
0928 02:51:23.672232 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:23.672300 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":48,"threads_started":1}
W20260928 02:51:23.673193 27516 catalog_manager.cc:4584] TS f9fb438fb3aa44d28cda797cb980a892 (127.19.139.129:38507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0970f6e7bf743609fac98bb3d22e34b: Network error: Client connection negotiation failed: client connection to 127.19.139.129:38507: connect: Connection refused (error 111)
I20260928 02:51:23.678766 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 27884
I20260928 02:51:23.687907 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 27500
2026-09-28T02:51:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3276 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[28145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[28145](info): set up 2 sockets
Sep 28 02:51:23 dist-test-slave-5vc0 krb5kdc[28145](info): commencing operation
krb5kdc: starting...
W20260928 02:51:24.648630 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:24 dist-test-slave-5vc0 krb5kdc[28145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:24.767674 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41297
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:41297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:24.876303 28158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:24.876605 28158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:24.876667 28158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:24.880762 28158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:24.880831 28158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:24.880856 28158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:24.880879 28158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:24.880908 28158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:24.886320 28158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:41297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:24.887419 28158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:24.888502 28158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:24.896687 28163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:24.896687 28166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:24.896678 28164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:24.897084 28158 server_base.cc:1034] running on GCE node
I20260928 02:51:24.897647 28158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:24.898774 28158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:24.899930 28158 hybrid_clock.cc:648] HybridClock initialized: now 1790563884899913 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:51:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1216 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:24 dist-test-slave-5vc0 krb5kdc[28171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:24 dist-test-slave-5vc0 krb5kdc[28171](info): set up 2 sockets
Sep 28 02:51:24 dist-test-slave-5vc0 krb5kdc[28171](info): commencing operation
krb5kdc: starting...
W20260928 02:51:25.864645 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:25 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:25.983078 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42487
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:26.091325 28184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:26.091637 28184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:26.091696 28184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:26.095764 28184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:26.095844 28184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:26.095870 28184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:26.095889 28184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:26.095908 28184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:26.101747 28184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:26.102850 28184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:26.103935 28184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:26.112134 28192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:26.112198 28184 server_base.cc:1034] running on GCE node
W20260928 02:51:26.112139 28189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:26.112254 28190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:26.112982 28184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:26.114064 28184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:26.115218 28184 hybrid_clock.cc:648] HybridClock initialized: now 1790563886115202 us; error 41 us; skew 500 ppm
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:26.121639 28184 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:26.122622 28184 webserver.cc:460] Webserver started at http://127.19.139.190:42891/ using document root <none> and password file <none>
I20260928 02:51:26.123191 28184 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:26.123260 28184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:26.123469 28184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:26.125589 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "04ba21327aca256859b922c4c75f3263"
server_key_iv: "400e3dfdbfabfdb7c236e65dd7665ba4"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.126164 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "04ba21327aca256859b922c4c75f3263"
server_key_iv: "400e3dfdbfabfdb7c236e65dd7665ba4"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.129931 28184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:26.132246 28199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.133481 28184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:26.133621 28184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "04ba21327aca256859b922c4c75f3263"
server_key_iv: "400e3dfdbfabfdb7c236e65dd7665ba4"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.133719 28184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:26.154569 28184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:26.157733 28184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:26.157917 28184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:26.164861 28184 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:42487
I20260928 02:51:26.164891 28251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:42487 every 8 connection(s)
I20260928 02:51:26.165832 28184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:26.168648 28252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:26.169224 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28184
I20260928 02:51:26.169333 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:26.169611 20014 external_mini_cluster.cc:1428] Setting key 2e900b1850e00f42739308eeed751849
I20260928 02:51:26.174518 28252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Bootstrap starting.
I20260928 02:51:26.176792 28252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:26.177492 28252 log.cc:824] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:26.179414 28252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: No bootstrap required, opened a new log
I20260928 02:51:26.182844 28252 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:26.183071 28252 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:26.183126 28252 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe2cfcbbc9f472e96e3bf27b60bcc0c, State: Initialized, Role: FOLLOWER
I20260928 02:51:26.183267 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.171021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.19.139.190:42487)
0928 02:51:26.171501 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:26.171515 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:26.171551 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:51:26.172238 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:26.172258 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:26.172604 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:26.172922 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:26.173714 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.174713 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.175556 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.176041 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.176223 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.178690 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:26.178696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:26.178709 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:26.178736 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:26.180341 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.180746 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.180751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.180756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.180880 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.181241 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.181244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.181246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.181671 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:26.181883 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:26.182141 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:51:26.182426 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260928 02:51:26.183594 28252 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:26.183737 28252 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:26.183799 28252 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:26.183892 28252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:26.185019 28252 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:26.185381 28252 leader_election.cc:302] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe2cfcbbc9f472e96e3bf27b60bcc0c; no voters:
I20260928 02:51:26.185711 28252 leader_election.cc:288] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:26.185757 28257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:26.185953 28257 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 LEADER]: Becoming Leader. State: Replica: abe2cfcbbc9f472e96e3bf27b60bcc0c, State: Running, Role: LEADER
I20260928 02:51:26.186239 28257 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:26.186980 28252 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:26.187866 28258 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe2cfcbbc9f472e96e3bf27b60bcc0c. Latest consensus state: current_term: 1 leader_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } } }
I20260928 02:51:26.187989 28258 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:26.188279 28259 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } } }
I20260928 02:51:26.188364 28259 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:26.189004 28266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:26.191510 28266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:26.196936 28266 catalog_manager.cc:1347] Generated new cluster ID: 40a7a0fd54bd461b9d2a605decca7c7b
I20260928 02:51:26.197062 28266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:26.212080 28266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:26.213092 28266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:26.224695 28266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Generated new TSK 0
I20260928 02:51:26.225425 28266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:26.255183 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42487
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:26.368607 28278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:26.368877 28278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:26.368921 28278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:26.373113 28278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:26.373184 28278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:26.373329 28278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:26.379101 28278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:26.380252 28278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:26.381311 28278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:26.390934 28286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:26.390944 28283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:26.390995 28284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:26.391178 28278 server_base.cc:1034] running on GCE node
I20260928 02:51:26.391605 28278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:26.392335 28278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:26.393494 28278 hybrid_clock.cc:648] HybridClock initialized: now 1790563886393474 us; error 33 us; skew 500 ppm
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:26.396234 28278 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:26.397166 28278 webserver.cc:460] Webserver started at http://127.19.139.129:46459/ using document root <none> and password file <none>
I20260928 02:51:26.397724 28278 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:26.397794 28278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:26.397987 28278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:26.400031 28278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "20100b1d709840d080b91b43d8b673d8"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "f9f0670a166288eda96214f22d1ef294"
server_key_iv: "acd3f07e4693eeca4e06ad9da55a4595"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.400665 28278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "20100b1d709840d080b91b43d8b673d8"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "f9f0670a166288eda96214f22d1ef294"
server_key_iv: "acd3f07e4693eeca4e06ad9da55a4595"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.404332 28278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:26.406690 28293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.407889 28278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:26.408006 28278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "20100b1d709840d080b91b43d8b673d8"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "f9f0670a166288eda96214f22d1ef294"
server_key_iv: "acd3f07e4693eeca4e06ad9da55a4595"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.408102 28278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:26.422225 28278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:26.425148 28278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:26.425330 28278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:26.425906 28278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:26.426726 28278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:26.426774 28278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.426826 28278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:26.426854 28278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.435199 28278 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:40809
I20260928 02:51:26.435225 28406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:40809 every 8 connection(s)
I20260928 02:51:26.436120 28278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:26.441468 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28278
I20260928 02:51:26.441599 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:26.441850 20014 external_mini_cluster.cc:1428] Setting key d3da4d203c48a2c783483ed80734d8be
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:26.449847 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.438051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:47539 (local address 127.19.139.190:42487)
0928 02:51:26.438182 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:26.438186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:26.439276 (+ 1090us) server_negotiation.cc:407] Connection header received
0928 02:51:26.440143 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:26.440146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:26.440227 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:26.440311 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:26.441506 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.442124 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.443474 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.443709 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.443797 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.446262 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:26.446265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:26.446267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:26.446291 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:26.447763 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.448323 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.448327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.448329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.448408 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.448741 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.448745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.448746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.448940 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:26.449029 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:26.449547 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 02:51:26.449679 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:26.450748 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.438458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:47539)
0928 02:51:26.439128 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:26.439171 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:26.439952 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:26.440481 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:26.440489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:26.440850 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:26.441347 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:26.441361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.442251 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.442254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:26.443348 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:26.443358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.443852 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.443856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:26.444031 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.444625 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:26.444642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:26.446098 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:26.447877 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:26.447884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:26.447901 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:26.448233 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:26.448498 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:26.448501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:26.448503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:26.448653 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:26.449041 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:26.449048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:26.449313 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:51:26.449493 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 02:51:26.449846 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":115,"threads_started":1}
I20260928 02:51:26.452242 28407 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42487
I20260928 02:51:26.452492 28407 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:51:26.453037 28407 heartbeater.cc:511] Master 127.19.139.190:42487 requested a full tablet report, sending...
I20260928 02:51:26.454804 28216 ts_manager.cc:194] Registered new tserver with Master: 20100b1d709840d080b91b43d8b673d8 (127.19.139.129:40809)
I20260928 02:51:26.456012 28216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:47539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:26.468000 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42487
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:26.576305 28412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:26.576597 28412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:26.576656 28412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:26.580739 28412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:26.580804 28412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:26.580888 28412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:26.586217 28412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:26.587284 28412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:26.588338 28412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:26.597849 28418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:26.597818 28417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:26.597827 28420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:26.598205 28412 server_base.cc:1034] running on GCE node
I20260928 02:51:26.598650 28412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:26.599277 28412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:26.600466 28412 hybrid_clock.cc:648] HybridClock initialized: now 1790563886600441 us; error 36 us; skew 500 ppm
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:26.603189 28412 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:26.604127 28412 webserver.cc:460] Webserver started at http://127.19.139.130:37539/ using document root <none> and password file <none>
I20260928 02:51:26.604720 28412 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:26.604789 28412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:26.604997 28412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:26.607089 28412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "e57b3ff2617f494eaa08babd445bdef5"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "02503dd5e50b0e4161889ccf444144c8"
server_key_iv: "114cda1f1c7f4520b091054f05dba0ee"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.607650 28412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "e57b3ff2617f494eaa08babd445bdef5"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "02503dd5e50b0e4161889ccf444144c8"
server_key_iv: "114cda1f1c7f4520b091054f05dba0ee"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.611532 28412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:26.614087 28427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.615336 28412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:51:26.615501 28412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "e57b3ff2617f494eaa08babd445bdef5"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "02503dd5e50b0e4161889ccf444144c8"
server_key_iv: "114cda1f1c7f4520b091054f05dba0ee"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.615607 28412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:26.630534 28412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:26.633832 28412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:26.634014 28412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:26.634593 28412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:26.635404 28412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:26.635461 28412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.635514 28412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:26.635538 28412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.644062 28412 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:40989
I20260928 02:51:26.644131 28540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:40989 every 8 connection(s)
I20260928 02:51:26.644992 28412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:26.653939 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28412
I20260928 02:51:26.654072 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:26.654366 20014 external_mini_cluster.cc:1428] Setting key 287a17ffcf21246b4ba2b6e56e6b6ee2
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:26.658914 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.646763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:58241 (local address 127.19.139.190:42487)
0928 02:51:26.646901 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:26.646905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:26.648036 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 02:51:26.648952 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:26.648956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:26.649011 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:26.649116 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:26.650347 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.650878 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.652054 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.652311 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.652406 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.655192 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:26.655196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:26.655198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:26.655223 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:26.656802 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.657406 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.657409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.657410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.657490 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.657848 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.657851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.657852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.658037 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:26.658122 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:26.658635 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 02:51:26.658753 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:51:26.659806 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.647186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:58241)
0928 02:51:26.647872 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:26.647916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:26.648748 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:26.649322 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:26.649329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:26.649709 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:26.650199 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:26.650214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.651001 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.651004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:26.651926 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:26.651937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.652459 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.652463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:26.652649 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.653293 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:26.653310 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:26.654990 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:26.656923 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:26.656930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:26.656947 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:26.657309 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:26.657598 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:26.657601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:26.657604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:26.657755 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:26.658136 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:26.658143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:26.658406 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 02:51:26.658580 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:51:26.658928 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":118,"threads_started":1}
I20260928 02:51:26.661056 28541 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42487
I20260928 02:51:26.661288 28541 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:26.661767 28541 heartbeater.cc:511] Master 127.19.139.190:42487 requested a full tablet report, sending...
I20260928 02:51:26.662801 28216 ts_manager.cc:194] Registered new tserver with Master: e57b3ff2617f494eaa08babd445bdef5 (127.19.139.130:40989)
I20260928 02:51:26.663343 28216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:58241
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:26.679435 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42487
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:26.791870 28546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:26.792147 28546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:26.792233 28546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:26.796314 28546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:26.796365 28546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:26.796435 28546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:26.801694 28546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:26.802745 28546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:26.803771 28546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:26.813120 28554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:26.813148 28551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:26.813253 28546 server_base.cc:1034] running on GCE node
W20260928 02:51:26.813428 28552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:26.813831 28546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:26.814476 28546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:26.815631 28546 hybrid_clock.cc:648] HybridClock initialized: now 1790563886815611 us; error 36 us; skew 500 ppm
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:26.818444 28546 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:26.819391 28546 webserver.cc:460] Webserver started at http://127.19.139.131:34805/ using document root <none> and password file <none>
I20260928 02:51:26.819954 28546 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:26.820016 28546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:26.820246 28546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:26.822295 28546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "f6e110f6af6f4a55af9124601660bdaa"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "c06385942b4db443ee4e65080f5cbde8"
server_key_iv: "7898d26f89e10a353393066624dd5863"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.822827 28546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "f6e110f6af6f4a55af9124601660bdaa"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "c06385942b4db443ee4e65080f5cbde8"
server_key_iv: "7898d26f89e10a353393066624dd5863"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.826566 28546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:51:26.829070 28561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.830324 28546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:26.830433 28546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "f6e110f6af6f4a55af9124601660bdaa"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "c06385942b4db443ee4e65080f5cbde8"
server_key_iv: "7898d26f89e10a353393066624dd5863"
server_key_version: "encryptionkey@0"
I20260928 02:51:26.830551 28546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:26.850136 28546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:26.853542 28546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:26.853721 28546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:26.854316 28546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:26.855202 28546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:26.855250 28546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.855289 28546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:26.855304 28546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:26.863569 28546 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:36231
I20260928 02:51:26.863675 28674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:36231 every 8 connection(s)
I20260928 02:51:26.864568 28546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:26.865253 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28546
I20260928 02:51:26.865358 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:26.865604 20014 external_mini_cluster.cc:1428] Setting key ea49afbe01679e69c4644f22257697c2
Sep 28 02:51:26 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563886, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:26.877441 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.866377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:47985 (local address 127.19.139.190:42487)
0928 02:51:26.866497 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:26.866501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:26.868007 (+ 1506us) server_negotiation.cc:407] Connection header received
0928 02:51:26.868882 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:26.868885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:26.868928 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:26.868995 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:26.870074 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.870604 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.871692 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.871905 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.871984 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.874415 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:26.874418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:26.874420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:26.874446 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:26.875622 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.876070 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.876072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.876073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.876144 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:26.876434 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:26.876436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:26.876437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:26.876639 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:26.876715 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:26.877215 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 02:51:26.877314 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:51:26.878348 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:26.866918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:47985)
0928 02:51:26.867884 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:26.867926 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:26.868714 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:26.869137 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:26.869144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:26.869490 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:26.869957 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:26.869970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.870692 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.870694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:26.871583 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:26.871597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:26.872019 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:26.872022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:26.872221 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:26.872800 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:26.872817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:26.874254 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:26.875718 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:26.875723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:26.875736 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:26.875983 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:26.876232 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:26.876234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:26.876236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:26.876369 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:26.876755 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:26.876761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:26.877013 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:51:26.877188 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:51:26.877471 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":128,"threads_started":1}
I20260928 02:51:26.879599 28675 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42487
I20260928 02:51:26.879829 28675 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:26.880338 28675 heartbeater.cc:511] Master 127.19.139.190:42487 requested a full tablet report, sending...
I20260928 02:51:26.881281 28215 ts_manager.cc:194] Registered new tserver with Master: f6e110f6af6f4a55af9124601660bdaa (127.19.139.131:36231)
I20260928 02:51:26.881837 28215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:47985
I20260928 02:51:26.891426 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab.
Entry for principal oryx/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab.
WARNING: no policy specified for oryx/127.19.139.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab.
Entry for principal oryx/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab.
WARNING: no policy specified for oryx/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab.
Entry for principal oryx/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab.
I20260928 02:51:26.962517 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28278
I20260928 02:51:26.969130 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28412
I20260928 02:51:26.973981 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28546
I20260928 02:51:26.978224 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28184
I20260928 02:51:26.983242 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42487
--webserver_interface=127.19.139.190
--webserver_port=42891
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:27.092520 28684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:27.092815 28684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:27.092875 28684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:27.096977 28684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:27.097062 28684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:27.097090 28684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:27.097110 28684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:27.097131 28684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:27.102746 28684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=42891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:27.103899 28684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:27.104992 28684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:27.113981 28690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:27.114310 28689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.114365 28684 server_base.cc:1034] running on GCE node
W20260928 02:51:27.114485 28692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.115074 28684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:27.116271 28684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:27.117434 28684 hybrid_clock.cc:648] HybridClock initialized: now 1790563887117412 us; error 38 us; skew 500 ppm
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563887, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:27.120289 28684 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:27.121243 28684 webserver.cc:460] Webserver started at http://127.19.139.190:42891/ using document root <none> and password file <none>
I20260928 02:51:27.121806 28684 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:27.121891 28684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:27.125823 28684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:27.128252 28699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.129604 28684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:27.129727 28684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "04ba21327aca256859b922c4c75f3263"
server_key_iv: "400e3dfdbfabfdb7c236e65dd7665ba4"
server_key_version: "encryptionkey@0"
I20260928 02:51:27.130281 28684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:27.147485 28684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:27.150640 28684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:27.150826 28684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:27.157966 28684 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:42487
I20260928 02:51:27.158033 28751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:42487 every 8 connection(s)
I20260928 02:51:27.159068 28684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:27.159312 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28684
I20260928 02:51:27.159754 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:40809
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:42487
--webserver_port=46459
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:51:27.164610 28752 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:51:27.167061 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Bootstrap starting.
I20260928 02:51:27.188942 28752 log.cc:824] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:27.197274 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:51:27.197701 28752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Bootstrap complete.
I20260928 02:51:27.202529 28752 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:27.202951 28752 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe2cfcbbc9f472e96e3bf27b60bcc0c, State: Initialized, Role: FOLLOWER
I20260928 02:51:27.203447 28752 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:27.203572 28752 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:27.203634 28752 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:27.203729 28752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:51:27.205768 28752 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:27.206094 28752 leader_election.cc:302] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe2cfcbbc9f472e96e3bf27b60bcc0c; no voters:
I20260928 02:51:27.206388 28752 leader_election.cc:288] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:51:27.206490 28756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:51:27.206687 28756 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [term 2 LEADER]: Becoming Leader. State: Replica: abe2cfcbbc9f472e96e3bf27b60bcc0c, State: Running, Role: LEADER
I20260928 02:51:27.206984 28756 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } }
I20260928 02:51:27.207647 28752 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:27.207962 28756 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe2cfcbbc9f472e96e3bf27b60bcc0c. Latest consensus state: current_term: 2 leader_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } } }
I20260928 02:51:27.208184 28756 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:27.208814 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe2cfcbbc9f472e96e3bf27b60bcc0c" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42487 } } }
I20260928 02:51:27.208985 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:27.209342 28760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:27.212596 28760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:27.213197 28760 catalog_manager.cc:1259] Loaded cluster ID: 40a7a0fd54bd461b9d2a605decca7c7b
I20260928 02:51:27.213251 28760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:27.215497 28760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:27.216148 28760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P abe2cfcbbc9f472e96e3bf27b60bcc0c: Loaded TSK: 0
I20260928 02:51:27.217525 28760 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:51:27.286870 28754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:27.287156 28754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:27.287200 28754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:27.291388 28754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:27.291476 28754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:27.291538 28754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:27.297009 28754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.129.keytab
--principal=oryx/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:40809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=46459
--tserver_master_addrs=127.19.139.190:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:27.298125 28754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:27.299192 28754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:27.310163 28778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:27.310151 28779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.310436 28754 server_base.cc:1034] running on GCE node
W20260928 02:51:27.310700 28781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.311141 28754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:27.311842 28754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:27.313055 28754 hybrid_clock.cc:648] HybridClock initialized: now 1790563887313007 us; error 69 us; skew 500 ppm
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563887, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:27.315801 28754 init.cc:377] Logged in from keytab as oryx/127.19.139.129@KRBTEST.COM (short username oryx)
I20260928 02:51:27.316856 28754 webserver.cc:460] Webserver started at http://127.19.139.129:46459/ using document root <none> and password file <none>
I20260928 02:51:27.317404 28754 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:27.317485 28754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:27.321532 28754 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:27.323607 28788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.324905 28754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:27.325011 28754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "20100b1d709840d080b91b43d8b673d8"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "f9f0670a166288eda96214f22d1ef294"
server_key_iv: "acd3f07e4693eeca4e06ad9da55a4595"
server_key_version: "encryptionkey@0"
I20260928 02:51:27.325563 28754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:27.341521 28754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:27.344718 28754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:27.344900 28754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:27.345475 28754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:27.346477 28754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:27.346534 28754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.346583 28754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:27.346598 28754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.355664 28754 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:40809
I20260928 02:51:27.355695 28901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:40809 every 8 connection(s)
I20260928 02:51:27.356734 28754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.366978 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28754
I20260928 02:51:27.367393 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:40989
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:42487
--webserver_port=37539
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:51:27.369505 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.358868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:56951 (local address 127.19.139.190:42487)
0928 02:51:27.359347 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.359367 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.359796 (+ 429us) server_negotiation.cc:407] Connection header received
0928 02:51:27.360872 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.360883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.361180 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.361485 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.362648 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.363628 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.364797 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.365269 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.365418 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.368746 (+ 3328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:56951: BlockingRecv error: recv got EOF from 127.19.139.129:56951 (error 108)
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":114,"threads_started":1}
I20260928 02:51:27.369515 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.358989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:56951)
0928 02:51:27.359643 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.359684 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.360485 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.361584 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.361592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.362008 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.362487 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.362502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.363758 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.363761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.364683 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.364694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.365424 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.365429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.365607 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.366133 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.366150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.368500 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":116,"threads_started":1}
W20260928 02:51:27.370078 28902 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:42487 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.139.190:42487: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.375315 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.370547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:49967)
0928 02:51:27.370656 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.370671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.370753 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.370979 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.370983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.371131 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.371226 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.371231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.371935 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.371938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.372518 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.372534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.372947 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.372951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.373039 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.373522 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.373560 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.375188 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:51:27.375424 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.370616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:49967 (local address 127.19.139.190:42487)
0928 02:51:27.370712 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.370716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.370729 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:27.370842 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.370845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.370891 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.370955 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.371329 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.371838 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.372628 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.372854 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.372936 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.375327 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:49967: BlockingRecv error: recv got EOF from 127.19.139.129:49967 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.380393 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.375843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:33139)
0928 02:51:27.375933 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.375946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.376016 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.376300 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.376303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.376436 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.376503 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.376507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.377189 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.377192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.377698 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.377705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.378105 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.378108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.378181 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.378616 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.378627 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.380260 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:51:27.380601 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.375942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:33139 (local address 127.19.139.190:42487)
0928 02:51:27.376082 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.376086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.376098 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:51:27.376130 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.376131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.376201 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.376264 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.376608 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.377097 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.377795 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.378007 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.378094 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.380378 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:33139: BlockingRecv error: recv got EOF from 127.19.139.129:33139 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:51:27.380622 28902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:51:27.476979 28906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:27.477248 28906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:27.477305 28906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:27.481323 28906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:27.481397 28906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:27.481478 28906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:27.486855 28906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.130.keytab
--principal=oryx/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:40989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=37539
--tserver_master_addrs=127.19.139.190:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:27.487932 28906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:27.488976 28906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:27.498437 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:27.498404 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:27.498481 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.498553 28906 server_base.cc:1034] running on GCE node
I20260928 02:51:27.499076 28906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:27.499732 28906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:27.500916 28906 hybrid_clock.cc:648] HybridClock initialized: now 1790563887500888 us; error 42 us; skew 500 ppm
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563887, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:27.503707 28906 init.cc:377] Logged in from keytab as oryx/127.19.139.130@KRBTEST.COM (short username oryx)
I20260928 02:51:27.504648 28906 webserver.cc:460] Webserver started at http://127.19.139.130:37539/ using document root <none> and password file <none>
I20260928 02:51:27.505179 28906 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:27.505251 28906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:27.509091 28906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:27.511217 28921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.512495 28906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:27.512593 28906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "e57b3ff2617f494eaa08babd445bdef5"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "02503dd5e50b0e4161889ccf444144c8"
server_key_iv: "114cda1f1c7f4520b091054f05dba0ee"
server_key_version: "encryptionkey@0"
I20260928 02:51:27.513168 28906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:27.524235 28906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:27.527249 28906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:27.527406 28906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:27.527979 28906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:27.528800 28906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:27.528846 28906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.528884 28906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:27.528903 28906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.537631 28906 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:40989
I20260928 02:51:27.537664 29034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:40989 every 8 connection(s)
I20260928 02:51:27.538657 28906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:27.543690 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 28906
I20260928 02:51:27.544198 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:36231
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:42487
--webserver_port=34805
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:51:27.550199 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.540501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:48841 (local address 127.19.139.190:42487)
0928 02:51:27.540646 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.540651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.541739 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 02:51:27.542559 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.542562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.542610 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.542682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.544050 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.544761 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.546150 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.546477 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.546588 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.550054 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:48841: BlockingRecv error: recv got EOF from 127.19.139.130:48841 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:51:27.550693 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.540942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:48841)
0928 02:51:27.541594 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.541637 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.542383 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.542843 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.542850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.543213 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.543898 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.543920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.544892 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.544896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.546018 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.546028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.546636 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.546639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.546783 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.547340 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.547355 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.549857 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":111,"threads_started":1}
W20260928 02:51:27.551196 29035 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:42487 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.139.190:42487: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.556375 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.551645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:47817)
0928 02:51:27.551741 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.551754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.551830 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.552064 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.552066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.552250 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.552340 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.552346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.553029 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.553033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.553577 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.553585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.553974 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.553978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.554063 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.554487 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.554498 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.556251 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:51:27.556475 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.551747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:47817 (local address 127.19.139.190:42487)
0928 02:51:27.551877 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.551881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.551893 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:51:27.551930 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.551932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.551968 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.552039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.552443 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.552934 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.553678 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.553884 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.553965 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.556375 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:47817: BlockingRecv error: recv got EOF from 127.19.139.130:47817 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.561914 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.557043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:37611)
0928 02:51:27.557195 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.557207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.557265 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.557471 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.557473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.557607 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.557677 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.557682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.558648 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.558650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.559150 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.559156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.559650 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.559653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.559706 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.560116 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.560127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.561817 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 02:51:27.562091 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.557138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:37611 (local address 127.19.139.190:42487)
0928 02:51:27.557248 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.557252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.557269 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:51:27.557332 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.557336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.557378 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.557464 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.557769 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.558553 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.559235 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.559552 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.559659 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.561977 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:37611: BlockingRecv error: recv got EOF from 127.19.139.130:37611 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 02:51:27.562268 29035 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:51:27.655813 29038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:27.656109 29038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:27.656181 29038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:27.660240 29038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:27.660303 29038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:27.660370 29038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:27.665706 29038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/krb5kdc/oryx_127.19.139.131.keytab
--principal=oryx/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:36231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=34805
--tserver_master_addrs=127.19.139.190:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:27.666795 29038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:27.667798 29038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:27.677345 29046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.677398 29038 server_base.cc:1034] running on GCE node
W20260928 02:51:27.677341 29043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:27.677348 29044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:27.677953 29038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:27.678611 29038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:27.679759 29038 hybrid_clock.cc:648] HybridClock initialized: now 1790563887679741 us; error 29 us; skew 500 ppm
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563887, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:27.682355 29038 init.cc:377] Logged in from keytab as oryx/127.19.139.131@KRBTEST.COM (short username oryx)
I20260928 02:51:27.683218 29038 webserver.cc:460] Webserver started at http://127.19.139.131:34805/ using document root <none> and password file <none>
I20260928 02:51:27.683732 29038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:27.683799 29038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:27.687618 29038 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:27.689603 29053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.690902 29038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:51:27.691008 29038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "f6e110f6af6f4a55af9124601660bdaa"
format_stamp: "Formatted at 2026-09-28 02:51:26 on dist-test-slave-5vc0"
server_key: "c06385942b4db443ee4e65080f5cbde8"
server_key_iv: "7898d26f89e10a353393066624dd5863"
server_key_version: "encryptionkey@0"
I20260928 02:51:27.691565 29038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:27.714995 29038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:27.717938 29038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:27.718092 29038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:27.718732 29038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:27.719564 29038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:27.719610 29038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.719650 29038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:27.719663 29038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:27.728046 29038 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:36231
I20260928 02:51:27.728082 29166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:36231 every 8 connection(s)
I20260928 02:51:27.729148 29038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:27.730582 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29038
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.740885 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.731598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.19.139.190:42487)
0928 02:51:27.732300 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.732305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.732319 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:27.732364 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.732367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.732413 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.732503 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.733366 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.733925 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.735087 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.735298 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.735384 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.737050 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:27.737056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:27.737070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:27.737098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:27.738799 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:27.739264 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:27.739269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:27.739274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:27.739415 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:27.739683 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:27.739685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:27.739686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:27.740032 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:27.740260 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:27.740506 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:51:27.740757 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":71,"threads_started":1}
W20260928 02:51:27.741065 29170 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.731174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:57445 (local address 127.19.139.190:42487)
0928 02:51:27.731805 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.731808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.732527 (+ 719us) server_negotiation.cc:407] Connection header received
0928 02:51:27.733692 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.733695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.733743 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.733846 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.735315 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.735902 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.737389 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.737711 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.737805 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.740950 (+ 3145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:57445: BlockingRecv error: recv got EOF from 127.19.139.131:57445 (error 108)
Metrics: {"server-negotiator.queue_time_us":566}
I20260928 02:51:27.741729 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.731676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:57445)
0928 02:51:27.732396 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.732439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.733487 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.734038 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.734048 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.734536 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.735188 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.735204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.736019 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.736022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.737302 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.737311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.737859 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.737863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.738038 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.738481 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.738496 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.740850 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":116,"threads_started":1}
W20260928 02:51:27.742210 29167 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:42487 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.139.190:42487: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.747565 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.742640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:38281)
0928 02:51:27.742950 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.742964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.743046 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.743268 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.743271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.743425 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.743556 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.743564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.744308 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.744311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.744897 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.744905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.745317 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.745320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.745445 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.745840 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.745851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.747463 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
W20260928 02:51:27.747714 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.742738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:38281 (local address 127.19.139.190:42487)
0928 02:51:27.742829 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.742832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.743019 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:51:27.743121 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.743124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.743173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.743234 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.743661 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.744205 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.744996 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.745220 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.745288 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.747595 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:38281: BlockingRecv error: recv got EOF from 127.19.139.131:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:27 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:27.752424 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:27.748027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:52355)
0928 02:51:27.748138 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:27.748149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:27.748255 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:27.748439 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:27.748442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:27.748600 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:27.748669 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.748674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.749346 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.749349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.749858 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:27.749865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.750258 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.750261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:27.750334 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.750739 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:27.750750 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:27.752293 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:51:27.752525 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:27.748103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:52355 (local address 127.19.139.190:42487)
0928 02:51:27.748265 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:27.748269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:27.748281 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:51:27.748317 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:27.748320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:27.748357 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:27.748416 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:27.748771 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.749255 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.749956 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:27.750167 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:27.750249 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:27.752424 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:52355: BlockingRecv error: recv got EOF from 127.19.139.131:52355 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260928 02:51:27.752698 29167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:51:28 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:28 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:28.386184 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:28.381159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:50773)
0928 02:51:28.381402 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:28.381415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:28.381501 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:28.381720 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:28.381722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:28.381895 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:28.382012 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:28.382017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.382764 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.382767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:28.383373 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:28.383387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.383768 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.383770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:28.383837 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:28.384337 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:28.384348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:28.386068 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
W20260928 02:51:28.386337 29173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:28.381235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:50773 (local address 127.19.139.190:42487)
0928 02:51:28.381464 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:28.381467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:28.381479 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:51:28.381577 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:28.381579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:28.381621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:28.381697 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:28.382122 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.382663 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.383461 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.383689 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.383755 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:28.386209 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:50773: BlockingRecv error: recv got EOF from 127.19.139.129:50773 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
Sep 28 02:51:28 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:28 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:28.568074 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:28.562920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:36507)
0928 02:51:28.563197 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:28.563211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:28.563323 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:28.563564 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:28.563567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:28.563766 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:28.563856 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:28.563862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.564632 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.564635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:28.565208 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:28.565216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.565634 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.565638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:28.565719 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:28.566152 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:28.566163 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:28.567913 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
W20260928 02:51:28.568187 29173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:28.563025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:36507 (local address 127.19.139.190:42487)
0928 02:51:28.563169 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:28.563174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:28.563283 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:51:28.563414 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:28.563417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:28.563461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:28.563555 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:28.563968 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.564533 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.565319 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.565536 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.565618 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:28.568058 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:36507: BlockingRecv error: recv got EOF from 127.19.139.130:36507 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:51:28 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:28 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:28.758939 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:28.753268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:39277)
0928 02:51:28.753583 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:28.753598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:28.753725 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:28.753993 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:28.753996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:28.754189 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:28.754377 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:28.754386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.755157 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.755160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:28.755753 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:28.755762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.756234 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.756238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:28.756343 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:28.756824 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:28.756838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:28.758788 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
W20260928 02:51:28.759042 29173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:28.753383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:39277 (local address 127.19.139.190:42487)
0928 02:51:28.753565 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:28.753569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:28.753682 (+ 113us) server_negotiation.cc:407] Connection header received
0928 02:51:28.753822 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:28.753825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:28.753881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:28.753956 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:28.754506 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.755046 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.755875 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:28.756111 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:28.756228 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:28.758924 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:39277: BlockingRecv error: recv got EOF from 127.19.139.131:39277 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 02:51:29 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:29 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:29.391878 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:29.386893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:42235)
0928 02:51:29.387144 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:29.387161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:29.387259 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:29.387470 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:29.387472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:29.387669 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:29.387763 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:29.387769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.388455 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.388457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:29.389020 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:29.389028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.389447 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.389450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:29.389517 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:29.389997 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:29.390008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:29.391717 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
W20260928 02:51:29.391981 29177 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:29.386981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:42235 (local address 127.19.139.190:42487)
0928 02:51:29.387249 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:29.387252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:29.387264 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:51:29.387337 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:29.387340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:29.387382 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:29.387443 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:29.387848 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.388380 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.389111 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.389352 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.389424 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:29.391865 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:42235: BlockingRecv error: recv got EOF from 127.19.139.129:42235 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
Sep 28 02:51:29 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:29 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:29.574421 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:29.568998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:60801)
0928 02:51:29.569304 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:29.569322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:29.569451 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:29.569699 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:29.569702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:29.569890 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:29.569985 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:29.569991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.570769 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.570773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:29.571337 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:29.571345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.571789 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.571793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:29.571878 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:29.572470 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:29.572482 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:29.574283 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
W20260928 02:51:29.574513 29177 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:29.569156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:60801 (local address 127.19.139.190:42487)
0928 02:51:29.569317 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:29.569322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:29.569402 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:51:29.569541 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:29.569545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:29.569592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:29.569659 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:29.570129 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.570656 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.571457 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.571679 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.571764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:29.574418 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:60801: BlockingRecv error: recv got EOF from 127.19.139.130:60801 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 02:51:29 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:29 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:29.764885 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:29.759674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:33971)
0928 02:51:29.759927 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:29.759940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:29.760025 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:29.760269 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:29.760272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:29.760442 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:29.760564 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:29.760570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.761260 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.761262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:29.761830 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:29.761839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.762290 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.762293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:29.762404 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:29.762869 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:29.762881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:29.764728 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
W20260928 02:51:29.764986 29177 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:29.759828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:33971 (local address 127.19.139.190:42487)
0928 02:51:29.759968 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:29.759971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:29.759984 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:29.760109 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:29.760112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:29.760174 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:29.760233 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:29.760658 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.761176 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.761918 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:29.762144 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:29.762216 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:29.764883 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:33971: BlockingRecv error: recv got EOF from 127.19.139.131:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:51:30 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:30 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:30.398088 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:30.392744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:57245)
0928 02:51:30.393005 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:30.393018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:30.393122 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:30.393388 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:30.393391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:30.393594 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:30.393690 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:30.393697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.394444 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.394448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:30.395076 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:30.395087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.395518 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.395522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:30.395606 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:30.396037 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:30.396050 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:30.397958 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
W20260928 02:51:30.398252 29181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:30.392798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:57245 (local address 127.19.139.190:42487)
0928 02:51:30.393035 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:30.393039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:30.393072 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:51:30.393231 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:30.393235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:30.393283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:30.393356 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:30.393802 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.394342 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.395190 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.395418 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.395503 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:30.398115 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:57245: BlockingRecv error: recv got EOF from 127.19.139.129:57245 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
Sep 28 02:51:30 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:30 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:30.580508 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:30.575164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:55287)
0928 02:51:30.575437 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:30.575455 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:30.575559 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:30.575766 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:30.575767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:30.575940 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:30.576033 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:30.576039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.576799 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.576802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:30.577363 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:30.577372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.577822 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.577825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:30.577892 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:30.578352 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:30.578364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:30.580327 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
W20260928 02:51:30.580634 29181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:30.575267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:55287 (local address 127.19.139.190:42487)
0928 02:51:30.575413 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:30.575417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:30.575522 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:51:30.575630 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:30.575632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:30.575674 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:30.575739 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:30.576133 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.576698 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.577472 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.577716 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.577792 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:30.580510 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:55287: BlockingRecv error: recv got EOF from 127.19.139.130:55287 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:51:30 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:30 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:30.771052 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:30.765704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:39303)
0928 02:51:30.765994 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:30.766008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:30.766119 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:30.766366 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:30.766369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:30.766576 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:30.766671 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:30.766677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.767411 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.767415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:30.767998 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:30.768006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.768499 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.768503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:30.768582 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:30.769116 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:30.769128 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:30.770926 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
W20260928 02:51:30.771159 29181 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:30.765822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:39303 (local address 127.19.139.190:42487)
0928 02:51:30.765958 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:30.765963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:30.766076 (+ 113us) server_negotiation.cc:407] Connection header received
0928 02:51:30.766213 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:30.766216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:30.766262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:30.766327 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:30.766787 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.767306 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.768108 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:30.768387 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:30.768503 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:30.771059 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:39303: BlockingRecv error: recv got EOF from 127.19.139.131:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:51:31 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:31 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:31.410012 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:31.398906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:59681)
0928 02:51:31.399233 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:31.399247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:31.399352 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:31.399669 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:31.399672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:31.399883 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:31.399980 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:31.399987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.401033 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.401036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:31.401694 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:31.401705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.402126 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.402128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:31.402188 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:31.407470 (+ 5282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:31.407485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:31.409886 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
W20260928 02:51:31.410210 29185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:31.399046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:59681 (local address 127.19.139.190:42487)
0928 02:51:31.399270 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:31.399273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:31.399305 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:51:31.399470 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:31.399473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:31.399520 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:31.399601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:31.400139 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.400921 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.401800 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.402035 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.402120 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:31.410065 (+ 7945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:59681: BlockingRecv error: recv got EOF from 127.19.139.129:59681 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":59,"threads_started":1}
Sep 28 02:51:31 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:31 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:31.586284 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:31.581227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:59247)
0928 02:51:31.581486 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:31.581500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:31.581624 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:31.581834 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:31.581836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:31.582003 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:31.582106 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:31.582112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.582839 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.582842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:31.583400 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:31.583409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.583853 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.583856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:31.583913 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:31.584449 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:31.584460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:31.586167 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
W20260928 02:51:31.586452 29185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:31.581300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:59247 (local address 127.19.139.190:42487)
0928 02:51:31.581428 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:31.581431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:31.581575 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:51:31.581696 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:31.581698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:31.581743 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:31.581809 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:31.582193 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.582732 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.583514 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.583764 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.583844 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:31.586320 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:59247: BlockingRecv error: recv got EOF from 127.19.139.130:59247 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:51:31 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:31 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:31.777186 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:31.771810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:36769)
0928 02:51:31.772061 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:31.772074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:31.772196 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:31.772462 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:31.772465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:31.772659 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:31.772773 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:31.772780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.773506 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.773509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:31.774130 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:31.774139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.774627 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.774630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:31.774714 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:31.775246 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:31.775258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:31.777067 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
W20260928 02:51:31.777338 29185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:31.771873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:36769 (local address 127.19.139.190:42487)
0928 02:51:31.772002 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:31.772006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:31.772134 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:51:31.772276 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:31.772278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:31.772324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:31.772382 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:31.772876 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.773387 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.774266 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:31.774503 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:31.774592 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:31.777211 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:36769: BlockingRecv error: recv got EOF from 127.19.139.131:36769 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:51:32 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:32 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:32.416055 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:32.410829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:41675)
0928 02:51:32.411099 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:32.411113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:32.411201 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:32.411428 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:32.411430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:32.411610 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:32.411704 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:32.411709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.412443 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.412445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:32.413026 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:32.413035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.413459 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.413462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:32.413524 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:32.414093 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:32.414105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:32.415908 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
W20260928 02:51:32.416218 29189 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:32.410919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:41675 (local address 127.19.139.190:42487)
0928 02:51:32.411169 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:32.411173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:32.411187 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:32.411280 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:32.411282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:32.411329 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:32.411407 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:32.411790 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.412357 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.413140 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.413361 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.413436 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:32.416033 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:41675: BlockingRecv error: recv got EOF from 127.19.139.129:41675 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
Sep 28 02:51:32 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:32 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:32.592331 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:32.587086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:46885)
0928 02:51:32.587349 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:32.587364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:32.587476 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:32.587719 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:32.587721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:32.587898 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:32.587993 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:32.587998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.588827 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.588830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:32.589402 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:32.589410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.589839 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.589842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:32.589905 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:32.590429 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:32.590440 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:32.592212 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
W20260928 02:51:32.592567 29189 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:32.587278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:46885 (local address 127.19.139.190:42487)
0928 02:51:32.587384 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:32.587387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:32.587415 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:51:32.587579 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:32.587582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:32.587625 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:32.587690 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:32.588096 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.588701 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.589503 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.589747 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.589826 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:32.592445 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:46885: BlockingRecv error: recv got EOF from 127.19.139.130:46885 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 02:51:32 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:32 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:32.783557 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:32.777999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:43839)
0928 02:51:32.778281 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:32.778294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:32.778397 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:32.778698 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:32.778700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:32.778883 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:32.778984 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:32.778990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.779752 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.779756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:32.780431 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:32.780443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.781000 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.781004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:32.781128 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:32.781621 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:32.781632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:32.783407 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
W20260928 02:51:32.783741 29189 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:32.778132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:43839 (local address 127.19.139.190:42487)
0928 02:51:32.778311 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:32.778314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:32.778345 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:51:32.778475 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:32.778478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:32.778526 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:32.778623 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:32.779085 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.779625 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.780582 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:32.780819 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:32.780912 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:32.783612 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:43839: BlockingRecv error: recv got EOF from 127.19.139.131:43839 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 28 02:51:33 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:33 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:33.422502 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:33.417014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:49749)
0928 02:51:33.417266 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:33.417280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:33.417407 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:33.417771 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:33.417774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:33.417970 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:33.418074 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:33.418081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.419061 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.419064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:33.419661 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:33.419672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.420064 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.420067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:33.420126 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:33.420597 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:33.420608 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:33.422354 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
W20260928 02:51:33.422613 29193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:33.417083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:49749 (local address 127.19.139.190:42487)
0928 02:51:33.417299 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:33.417302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:33.417333 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:51:33.417564 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:33.417568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:33.417626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:33.417724 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:33.418188 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.418956 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.419767 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.419983 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.420057 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:33.422490 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:49749: BlockingRecv error: recv got EOF from 127.19.139.129:49749 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
Sep 28 02:51:33 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:33 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:33.598436 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:33.593219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:57207)
0928 02:51:33.593461 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:33.593474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:33.593579 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:33.593791 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:33.593793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:33.593970 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:33.594088 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:33.594094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.594915 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.594919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:33.595484 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:33.595494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.595925 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.595928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:33.596010 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:33.596509 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:33.596521 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:33.598318 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
W20260928 02:51:33.598608 29193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:33.593297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:57207 (local address 127.19.139.190:42487)
0928 02:51:33.593476 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:33.593481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:33.593543 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:51:33.593651 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:33.593654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:33.593697 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:33.593772 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:33.594188 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.594809 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.595597 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.595818 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.595917 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:33.598459 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:57207: BlockingRecv error: recv got EOF from 127.19.139.130:57207 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 02:51:33 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:33 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:33.789541 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:33.784320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:59765)
0928 02:51:33.784582 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:33.784597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:33.784694 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:33.784891 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:33.784893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:33.785074 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:33.785167 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:33.785173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.786104 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.786107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:33.786702 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:33.786710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.787149 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.787152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:33.787210 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:33.787686 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:33.787698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:33.789417 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
W20260928 02:51:33.789664 29193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:33.784360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:59765 (local address 127.19.139.190:42487)
0928 02:51:33.784473 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:33.784477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:33.784655 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:51:33.784756 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:33.784759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:33.784804 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:33.784867 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:33.785249 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.785981 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.786813 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:33.787038 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:33.787139 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:33.789545 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:59765: BlockingRecv error: recv got EOF from 127.19.139.131:59765 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:51:34 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:34 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:34.428081 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:34.423189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:45037)
0928 02:51:34.423461 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:34.423475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:34.423585 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:34.423807 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:34.423809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:34.423985 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:34.424085 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:34.424090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.424795 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.424798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:34.425345 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:34.425352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.425717 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.425720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:34.425785 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:34.426239 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:34.426249 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:34.427961 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":69,"threads_started":1}
W20260928 02:51:34.428274 29197 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:34.423278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:45037 (local address 127.19.139.190:42487)
0928 02:51:34.423501 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:34.423504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:34.423517 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:34.423659 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:34.423661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:34.423703 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:34.423779 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:34.424192 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.424705 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.425426 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.425635 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.425702 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:34.428109 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:45037: BlockingRecv error: recv got EOF from 127.19.139.129:45037 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
Sep 28 02:51:34 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:34 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:34.604857 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:34.599204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:47851)
0928 02:51:34.599584 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:34.599597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:34.599698 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:34.599949 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:34.599951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:34.600209 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:34.600310 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:34.600316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.601110 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.601114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:34.601695 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:34.601704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.602173 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.602176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:34.602244 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:34.602765 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:34.602777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:34.604733 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":96,"threads_started":1}
W20260928 02:51:34.604987 29197 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:34.599306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:47851 (local address 127.19.139.190:42487)
0928 02:51:34.599455 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:34.599459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:34.599695 (+ 236us) server_negotiation.cc:407] Connection header received
0928 02:51:34.599767 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:34.599769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:34.599824 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:34.599912 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:34.600423 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.600989 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.601821 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.602046 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.602140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:34.604880 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:47851: BlockingRecv error: recv got EOF from 127.19.139.130:47851 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:51:34 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:34 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:34.795701 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:34.790406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:56879)
0928 02:51:34.790732 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:34.790746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:34.790840 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:34.791053 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:34.791056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:34.791230 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:34.791323 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:34.791328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.792112 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.792115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:34.792718 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:34.792727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.793139 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.793141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:34.793200 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:34.793667 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:34.793678 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:34.795564 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
W20260928 02:51:34.795890 29197 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:34.790516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:56879 (local address 127.19.139.190:42487)
0928 02:51:34.790654 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:34.790658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:34.790793 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:51:34.790921 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:34.790923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:34.790967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:34.791025 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:34.791471 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.792003 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.792823 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:34.793049 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:34.793130 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:34.795718 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:56879: BlockingRecv error: recv got EOF from 127.19.139.131:56879 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 02:51:35 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:35 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:35.433984 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:35.428802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:57171)
0928 02:51:35.429092 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:35.429106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:35.429204 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:35.429463 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:35.429466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:35.429672 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:35.429770 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:35.429776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.430500 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.430503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:35.431101 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:35.431109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.431480 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.431483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:35.431545 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:35.432037 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:35.432050 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:35.433833 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
W20260928 02:51:35.434109 29201 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:35.428980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:57171 (local address 127.19.139.190:42487)
0928 02:51:35.429222 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:35.429225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:35.429239 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:35.429277 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:35.429279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:35.429321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:35.429394 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:35.429868 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.430409 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.431188 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.431397 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.431462 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:35.433977 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:57171: BlockingRecv error: recv got EOF from 127.19.139.129:57171 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
Sep 28 02:51:35 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:35 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:35.611241 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:35.605638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:33953)
0928 02:51:35.605939 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:35.605955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:35.606075 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:35.606313 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:35.606315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:35.606508 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:35.606601 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:35.606607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.607510 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.607514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:35.608140 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:35.608176 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.608645 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.608669 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:35.608749 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:35.609213 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:35.609224 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:35.611083 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
W20260928 02:51:35.611351 29201 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:35.605755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:33953 (local address 127.19.139.190:42487)
0928 02:51:35.605919 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:35.605924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:35.606026 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:51:35.606162 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:35.606166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:35.606211 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:35.606276 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:35.606715 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.607369 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.608275 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.608530 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.608614 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:35.611236 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:33953: BlockingRecv error: recv got EOF from 127.19.139.130:33953 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:51:35 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:35 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:35.801898 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:35.796485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:59811)
0928 02:51:35.796747 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:35.796760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:35.796865 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:35.797119 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:35.797122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:35.797300 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:35.797395 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:35.797401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.798115 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.798118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:35.798683 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:35.798692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.799216 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.799219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:35.799291 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:35.799840 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:35.799852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:35.801778 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
W20260928 02:51:35.802013 29201 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:35.796581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:59811 (local address 127.19.139.190:42487)
0928 02:51:35.796709 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:35.796713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:35.796828 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:51:35.796962 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:35.796964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:35.797011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:35.797102 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:35.797485 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.798010 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.798809 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:35.799104 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:35.799211 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:35.801915 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:59811: BlockingRecv error: recv got EOF from 127.19.139.131:59811 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 02:51:36 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:36 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:36.444314 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:36.434861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:46591)
0928 02:51:36.435118 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:36.435133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:36.435224 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:36.435450 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:36.435452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:36.435648 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:36.435746 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:36.435752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.436656 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.436660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:36.437247 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:36.437257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.437697 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.437700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:36.437768 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:36.442192 (+ 4424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:36.442206 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:36.444193 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20260928 02:51:36.444482 29205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:36.434941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:46591 (local address 127.19.139.190:42487)
0928 02:51:36.435202 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:36.435206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:36.435222 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:51:36.435294 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:36.435297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:36.435338 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:36.435435 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:36.435891 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.436529 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.437357 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.437576 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.437648 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:36.444341 (+ 6693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:46591: BlockingRecv error: recv got EOF from 127.19.139.129:46591 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":120,"threads_started":1}
Sep 28 02:51:36 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:36 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:36.617261 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:36.612081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:40913)
0928 02:51:36.612361 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:36.612375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:36.612481 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:36.612707 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:36.612709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:36.612886 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:36.612986 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:36.612992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.613677 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.613680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:36.614256 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:36.614265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.614652 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.614654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:36.614715 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:36.615169 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:36.615180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:36.617087 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
W20260928 02:51:36.617343 29205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:36.612222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:40913 (local address 127.19.139.190:42487)
0928 02:51:36.612369 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:36.612373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:36.612439 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:51:36.612572 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:36.612574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:36.612619 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:36.612683 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:36.613083 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.613590 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.614349 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.614558 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.614625 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:36.617242 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:40913: BlockingRecv error: recv got EOF from 127.19.139.130:40913 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:51:36 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:36 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:36.808389 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:36.802745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:45563)
0928 02:51:36.803029 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:36.803043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:36.803154 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:36.803378 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:36.803381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:36.803553 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:36.803645 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:36.803650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.804512 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.804515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:36.805220 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:36.805234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.805761 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.805764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:36.805830 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:36.806312 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:36.806325 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:36.808271 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
W20260928 02:51:36.808561 29205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:36.802884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:45563 (local address 127.19.139.190:42487)
0928 02:51:36.803030 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:36.803034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:36.803152 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:51:36.803223 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:36.803226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:36.803275 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:36.803355 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:36.803775 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.804397 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.805330 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:36.805654 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:36.805749 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:36.808409 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:45563: BlockingRecv error: recv got EOF from 127.19.139.131:45563 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:51:37 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:37 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:37.450277 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:37.445117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:57943)
0928 02:51:37.445364 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:37.445377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:37.445466 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:37.445780 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:37.445783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:37.445954 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:37.446054 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:37.446060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.446785 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.446787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:37.447337 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:37.447344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.447809 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.447812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:37.447881 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:37.448372 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:37.448384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:37.450161 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
W20260928 02:51:37.450412 29209 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:37.445178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:57943 (local address 127.19.139.190:42487)
0928 02:51:37.445406 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:37.445409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:37.445424 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:51:37.445553 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:37.445555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:37.445604 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:37.445679 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:37.446154 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.446701 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.447439 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.447683 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.447758 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:37.450293 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:57943: BlockingRecv error: recv got EOF from 127.19.139.129:57943 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
Sep 28 02:51:37 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:37 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:37.623409 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:37.618141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:51327)
0928 02:51:37.618399 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:37.618415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:37.618529 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:37.618780 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:37.618783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:37.618986 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:37.619080 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:37.619085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.619833 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.619836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:37.620466 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:37.620475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.620906 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.620909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:37.620991 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:37.621442 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:37.621454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:37.623281 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
W20260928 02:51:37.623544 29209 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:37.618216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:51327 (local address 127.19.139.190:42487)
0928 02:51:37.618370 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:37.618373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:37.618494 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:51:37.618632 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:37.618635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:37.618683 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:37.618745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:37.619191 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.619730 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.620576 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.620800 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.620887 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:37.623423 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:51327: BlockingRecv error: recv got EOF from 127.19.139.130:51327 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:51:37 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:37 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:37.814332 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:37.809202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:43515)
0928 02:51:37.809476 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:37.809490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:37.809612 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:37.809864 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:37.809867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:37.810073 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:37.810170 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:37.810176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.810922 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.810925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:37.811495 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:37.811503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.811935 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.811938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:37.812018 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:37.812472 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:37.812483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:37.814196 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
W20260928 02:51:37.814456 29209 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:37.809317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:43515 (local address 127.19.139.190:42487)
0928 02:51:37.809469 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:37.809473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:37.809552 (+ 79us) server_negotiation.cc:407] Connection header received
0928 02:51:37.809707 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:37.809710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:37.809762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:37.809828 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:37.810291 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.810821 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.811608 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:37.811838 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:37.811922 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:37.814336 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:43515: BlockingRecv error: recv got EOF from 127.19.139.131:43515 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:51:38 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:38 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:38.456523 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:38.451091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:42409)
0928 02:51:38.451358 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:38.451371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:38.451476 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:38.451768 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:38.451771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:38.451967 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:38.452068 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:38.452074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.452964 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.452966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:38.453576 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:38.453588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.453997 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.453999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:38.454057 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:38.454529 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:38.454543 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:38.456389 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
W20260928 02:51:38.456631 29213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:38.451219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:42409 (local address 127.19.139.190:42487)
0928 02:51:38.451453 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:38.451457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:38.451472 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:51:38.451587 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:38.451590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:38.451646 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:38.451739 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:38.452226 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.452833 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.453690 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.453912 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.453986 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:38.456513 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:42409: BlockingRecv error: recv got EOF from 127.19.139.129:42409 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
Sep 28 02:51:38 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:38 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:38.631604 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:38.624420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:37735)
0928 02:51:38.624807 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:38.624823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:38.624950 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:38.625236 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:38.625239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:38.625469 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:38.625594 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:38.625602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.626607 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.626611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:38.627408 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:38.627420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.627964 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.627969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:38.628071 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:38.628706 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:38.628722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:38.631450 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":100,"threads_started":1}
W20260928 02:51:38.631698 29213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:38.624530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:37735 (local address 127.19.139.190:42487)
0928 02:51:38.624678 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:38.624683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:38.624897 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:51:38.625043 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:38.625047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:38.625101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:38.625197 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:38.625723 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.626483 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.627503 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.627834 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.627942 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:38.631593 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:37735: BlockingRecv error: recv got EOF from 127.19.139.130:37735 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 02:51:38 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:38 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:38.820904 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:38.815158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:52665)
0928 02:51:38.815454 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:38.815469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:38.815590 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:38.815880 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:38.815883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:38.816089 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:38.816246 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:38.816252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.817082 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.817085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:38.817643 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:38.817653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.818077 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.818081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:38.818164 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:38.818695 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:38.818708 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:38.820751 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
W20260928 02:51:38.820986 29213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:38.815290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:52665 (local address 127.19.139.190:42487)
0928 02:51:38.815433 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:38.815438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:38.815540 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:51:38.815705 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:38.815709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:38.815770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:38.815840 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:38.816358 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.816969 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.817751 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:38.817967 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:38.818052 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:38.820887 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:52665: BlockingRecv error: recv got EOF from 127.19.139.131:52665 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:51:39 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:39 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:39.462219 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:39.457278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:41151)
0928 02:51:39.457535 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:39.457548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:39.457666 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:39.457882 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:39.457884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:39.458051 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:39.458147 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:39.458152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.458833 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.458836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:39.459390 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:39.459396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.459783 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.459786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:39.459841 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:39.460387 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:39.460400 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:39.462104 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
W20260928 02:51:39.462400 29217 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:39.457345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:41151 (local address 127.19.139.190:42487)
0928 02:51:39.457571 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:39.457574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:39.457620 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:51:39.457741 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:39.457744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:39.457786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:39.457857 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:39.458231 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.458753 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.459477 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.459703 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.459776 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:39.462251 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:41151: BlockingRecv error: recv got EOF from 127.19.139.129:41151 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":59,"threads_started":1}
Sep 28 02:51:39 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:39 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:39.637733 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:39.632412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:57045)
0928 02:51:39.632679 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:39.632694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:39.632817 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:39.633056 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:39.633059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:39.633280 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:39.633395 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:39.633402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.634209 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.634212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:39.634798 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:39.634807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.635244 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.635248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:39.635330 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:39.635800 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:39.635811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:39.637613 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
W20260928 02:51:39.637899 29217 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:39.632542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:57045 (local address 127.19.139.190:42487)
0928 02:51:39.632681 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:39.632685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:39.632772 (+ 87us) server_negotiation.cc:407] Connection header received
0928 02:51:39.632907 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:39.632911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:39.632957 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:39.633024 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:39.633510 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.634102 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.634911 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.635141 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.635228 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:39.637784 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:57045: BlockingRecv error: recv got EOF from 127.19.139.130:57045 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:51:39 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:39 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:39.827066 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:39.821724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:57037)
0928 02:51:39.821980 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:39.821993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:39.822085 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:39.822284 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:39.822286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:39.822459 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:39.822557 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:39.822563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.823514 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.823517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:39.824101 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:39.824112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.824609 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.824612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:39.824688 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:39.825154 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:39.825165 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:39.826943 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
W20260928 02:51:39.827250 29217 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:39.821766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:57037 (local address 127.19.139.190:42487)
0928 02:51:39.821872 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:39.821876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:39.822038 (+ 162us) server_negotiation.cc:407] Connection header received
0928 02:51:39.822149 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:39.822151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:39.822195 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:39.822257 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:39.822752 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.823377 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.824234 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:39.824474 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:39.824559 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:39.827122 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:57037: BlockingRecv error: recv got EOF from 127.19.139.131:57037 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 02:51:40 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:40 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:40.468184 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:40.463003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:47063)
0928 02:51:40.463268 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:40.463282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:40.463375 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:40.463613 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:40.463615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:40.463786 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:40.463878 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:40.463884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.464629 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.464632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:40.465185 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:40.465194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.465637 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.465639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:40.465698 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:40.466168 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:40.466180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:40.468033 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
W20260928 02:51:40.468434 29221 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:40.463101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:47063 (local address 127.19.139.190:42487)
0928 02:51:40.463324 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:40.463327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:40.463340 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:40.463458 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:40.463461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:40.463504 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:40.463584 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:40.463966 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.464514 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.465281 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.465527 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.465605 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:40.468258 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:47063: BlockingRecv error: recv got EOF from 127.19.139.129:47063 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
Sep 28 02:51:40 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:40 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:40.643591 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:40.638465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:50251)
0928 02:51:40.638746 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:40.638761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:40.638866 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:40.639074 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:40.639076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:40.639252 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:40.639349 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:40.639354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.640057 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.640061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:40.640694 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:40.640705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.641123 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.641125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:40.641185 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:40.641624 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:40.641635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:40.643428 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
W20260928 02:51:40.643672 29221 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:40.638648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:50251 (local address 127.19.139.190:42487)
0928 02:51:40.638780 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:40.638784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:40.638814 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:51:40.638938 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:40.638941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:40.638984 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:40.639046 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:40.639436 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.639943 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.640813 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.641031 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.641116 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:40.643571 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:50251: BlockingRecv error: recv got EOF from 127.19.139.130:50251 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:51:40 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:40 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:40.833002 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:40.827822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:40825)
0928 02:51:40.828098 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:40.828112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:40.828249 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:40.828548 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:40.828551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:40.828801 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:40.828899 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:40.828905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.829672 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.829674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:40.830268 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:40.830277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.830661 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.830664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:40.830726 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:40.831163 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:40.831174 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:40.832851 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
W20260928 02:51:40.833101 29221 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:40.827912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:40825 (local address 127.19.139.190:42487)
0928 02:51:40.828037 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:40.828040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:40.828190 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:51:40.828369 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:40.828373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:40.828434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:40.828521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:40.828999 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.829569 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.830354 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:40.830568 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:40.830640 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:40.832990 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:40825: BlockingRecv error: recv got EOF from 127.19.139.131:40825 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 02:51:41 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:41 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:41.479974 29224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:41.469187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:42749)
0928 02:51:41.469578 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:41.469595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:41.469704 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:41.469971 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:41.469973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:41.470164 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:41.470277 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:41.470286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.471039 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.471042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:41.471669 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:41.471678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.472066 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.472069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:41.472137 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:41.477656 (+ 5519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:41.477673 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:41.479809 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":79,"threads_started":1}
W20260928 02:51:41.480149 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:41.469272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:42749 (local address 127.19.139.190:42487)
0928 02:51:41.469578 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:41.469582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:41.469657 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:51:41.469791 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:41.469795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:41.469846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:41.469928 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:41.470396 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.470930 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.471761 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.471969 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.472035 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:41.480002 (+ 7967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:42749: BlockingRecv error: recv got EOF from 127.19.139.129:42749 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":71,"threads_started":1}
Sep 28 02:51:41 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:41 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:41.650229 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:41.644568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:52119)
0928 02:51:41.644853 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:41.644867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:41.644979 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:41.645214 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:41.645217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:41.645434 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:41.645546 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:41.645552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.646335 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.646339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:41.647176 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:41.647189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.647612 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.647615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:41.647677 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:41.648135 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:41.648146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:41.650068 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
W20260928 02:51:41.650319 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:41.644704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:52119 (local address 127.19.139.190:42487)
0928 02:51:41.644861 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:41.644865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:41.644930 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:51:41.645066 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:41.645069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:41.645112 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:41.645179 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:41.645644 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.646224 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.647274 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.647514 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.647617 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:41.650217 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:52119: BlockingRecv error: recv got EOF from 127.19.139.130:52119 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:51:41 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:41 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.131@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:41.839339 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:41.833870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.131:36211)
0928 02:51:41.834156 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:41.834172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:41.834279 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:41.834581 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:41.834584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:41.834768 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:41.834868 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:41.834874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.835755 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.835759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:41.836395 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:41.836405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.836871 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.836874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:41.836957 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:41.837440 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:41.837453 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:41.839182 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":98,"threads_started":1}
W20260928 02:51:41.839438 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:41.833968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:36211 (local address 127.19.139.190:42487)
0928 02:51:41.834108 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:41.834112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:41.834237 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:51:41.834359 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:41.834362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:41.834413 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:41.834494 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:41.835005 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.835633 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.836511 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:41.836764 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:41.836852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:41.839334 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.131:36211: BlockingRecv error: recv got EOF from 127.19.139.131:36211 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:51:42 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:42 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.129@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:42.488895 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:42.480943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.129:34137)
0928 02:51:42.481226 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:42.481243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:42.481338 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:42.481565 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:42.481567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:42.481738 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:42.481831 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:42.481836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.482529 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.482532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:42.483094 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:42.483100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.483468 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.483470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:42.483525 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:42.487078 (+ 3553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:42.487092 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:42.488769 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
W20260928 02:51:42.489028 29229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:42.481058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:34137 (local address 127.19.139.190:42487)
0928 02:51:42.481323 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:42.481326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:42.481340 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:51:42.481419 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:42.481422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:42.481463 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:42.481528 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:42.481916 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.482447 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.483175 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.483390 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.483459 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:42.488910 (+ 5451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.129:34137: BlockingRecv error: recv got EOF from 127.19.139.129:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
Sep 28 02:51:42 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:51:42 dist-test-slave-5vc0 krb5kdc[28171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.139.130@KRBTEST.COM for oryx/127.19.139.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:51:42.656519 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:42.651182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42487 (local address 127.19.139.130:38237)
0928 02:51:42.651465 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:42.651483 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:42.651593 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:42.651805 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:42.651807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:42.651988 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:42.652082 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:42.652089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.652850 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.652853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:42.653452 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:42.653462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.653963 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.653966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:42.654033 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:42.654531 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:42.654542 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:42.656368 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.139.190:42487: Server oryx/127.19.139.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
W20260928 02:51:42.656625 29229 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:42.651270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:38237 (local address 127.19.139.190:42487)
0928 02:51:42.651421 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:42.651426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:42.651559 (+ 133us) server_negotiation.cc:407] Connection header received
0928 02:51:42.651666 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:42.651668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:42.651712 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:42.651775 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:42.652188 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.652739 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.653587 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.653830 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.653922 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:42.656525 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.139.130:38237: BlockingRecv error: recv got EOF from 127.19.139.130:38237 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:51:42.731153 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28754
I20260928 02:51:42.735869 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28906
I20260928 02:51:42.740116 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29038
I20260928 02:51:42.744323 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 28684
2026-09-28T02:51:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17837 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:42Z Disabled control of system clock
I20260928 02:51:42.771993 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37259
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:51:42.879896 29234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:42.880198 29234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:42.880247 29234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:42.884286 29234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:42.884346 29234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:42.884361 29234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:42.884375 29234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:42.884390 29234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:42.889302 29234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:42.890396 29234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:42.891824 29234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:42.900451 29239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:42.900516 29234 server_base.cc:1034] running on GCE node
W20260928 02:51:42.900447 29240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:42.900447 29242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:42.901350 29234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:42.902489 29234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:42.903637 29234 hybrid_clock.cc:648] HybridClock initialized: now 1790563902903613 us; error 42 us; skew 500 ppm
I20260928 02:51:42.905472 29234 webserver.cc:460] Webserver started at http://127.19.139.190:33637/ using document root <none> and password file <none>
I20260928 02:51:42.906025 29234 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:42.906088 29234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:42.906286 29234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:42.908442 29234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "2e34f76eb409491eac71e33fe2fe6d89"
format_stamp: "Formatted at 2026-09-28 02:51:42 on dist-test-slave-5vc0"
server_key: "226f1c3461bd61a7a0a3a0070164fdcd"
server_key_iv: "dad0c93daa86b2d7ee1ca05c8b89089e"
server_key_version: "encryptionkey@0"
I20260928 02:51:42.909004 29234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "2e34f76eb409491eac71e33fe2fe6d89"
format_stamp: "Formatted at 2026-09-28 02:51:42 on dist-test-slave-5vc0"
server_key: "226f1c3461bd61a7a0a3a0070164fdcd"
server_key_iv: "dad0c93daa86b2d7ee1ca05c8b89089e"
server_key_version: "encryptionkey@0"
I20260928 02:51:42.912511 29234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:51:42.914793 29248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:42.915939 29234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:42.916044 29234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "2e34f76eb409491eac71e33fe2fe6d89"
format_stamp: "Formatted at 2026-09-28 02:51:42 on dist-test-slave-5vc0"
server_key: "226f1c3461bd61a7a0a3a0070164fdcd"
server_key_iv: "dad0c93daa86b2d7ee1ca05c8b89089e"
server_key_version: "encryptionkey@0"
I20260928 02:51:42.916149 29234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:42.944998 29234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:42.945730 29234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:42.945910 29234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:42.952658 29234 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:37259
I20260928 02:51:42.952687 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:37259 every 8 connection(s)
I20260928 02:51:42.953631 29234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:42.956902 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:42.957898 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29234
I20260928 02:51:42.958014 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:42.958324 20014 external_mini_cluster.cc:1428] Setting key 0845361e4b974b8d8a898a2d2b4ed7e7
I20260928 02:51:42.963399 29301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89: Bootstrap starting.
I20260928 02:51:42.965879 29301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:42.966629 29301 log.cc:824] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:42.967883 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:42.959841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.19.139.190:37259)
0928 02:51:42.960535 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:42.960549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:42.960615 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:51:42.961280 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:42.961298 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:42.961738 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:42.962066 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:42.962525 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.963573 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.964483 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:42.965006 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:42.965151 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:42.966023 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:42.966029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:42.966048 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:42.966451 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:42.966509 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:42.966529 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:42.966623 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:42.966760 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:51:42.967021 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":139,"threads_started":1}
I20260928 02:51:42.968745 29301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89: No bootstrap required, opened a new log
I20260928 02:51:42.972293 29301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e34f76eb409491eac71e33fe2fe6d89" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37259 } }
I20260928 02:51:42.972532 29301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:42.972589 29301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e34f76eb409491eac71e33fe2fe6d89, State: Initialized, Role: FOLLOWER
I20260928 02:51:42.972994 29301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e34f76eb409491eac71e33fe2fe6d89" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37259 } }
I20260928 02:51:42.973119 29301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:42.973176 29301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:42.973273 29301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:42.974176 29301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e34f76eb409491eac71e33fe2fe6d89" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37259 } }
I20260928 02:51:42.974499 29301 leader_election.cc:302] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e34f76eb409491eac71e33fe2fe6d89; no voters:
I20260928 02:51:42.974833 29301 leader_election.cc:288] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:42.974946 29306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:42.975124 29306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [term 1 LEADER]: Becoming Leader. State: Replica: 2e34f76eb409491eac71e33fe2fe6d89, State: Running, Role: LEADER
I20260928 02:51:42.975472 29306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e34f76eb409491eac71e33fe2fe6d89" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37259 } }
I20260928 02:51:42.976111 29301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:42.977131 29307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e34f76eb409491eac71e33fe2fe6d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e34f76eb409491eac71e33fe2fe6d89" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37259 } } }
I20260928 02:51:42.977269 29307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:42.977651 29315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:42.977607 29308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e34f76eb409491eac71e33fe2fe6d89. Latest consensus state: current_term: 1 leader_uuid: "2e34f76eb409491eac71e33fe2fe6d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e34f76eb409491eac71e33fe2fe6d89" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37259 } } }
I20260928 02:51:42.978284 29308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:42.980875 29315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:42.986575 29315 catalog_manager.cc:1347] Generated new cluster ID: 893dd35cd3f84ad9a60086cf9a12422c
I20260928 02:51:42.986650 29315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:42.997989 29315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:42.998984 29315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:43.006309 29315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e34f76eb409491eac71e33fe2fe6d89: Generated new TSK 0
I20260928 02:51:43.007061 29315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:43.016386 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37259
--builtin_ntp_servers=127.19.139.148:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:51:43.124053 29325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:43.124341 29325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:43.124385 29325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:43.128345 29325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:43.128397 29325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:43.128458 29325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:43.133133 29325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:43.134192 29325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:43.135557 29325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:43.145013 29331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:43.145074 29330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:43.145140 29325 server_base.cc:1034] running on GCE node
W20260928 02:51:43.145046 29333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:43.145630 29325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:43.146231 29325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:43.147399 29325 hybrid_clock.cc:648] HybridClock initialized: now 1790563903147385 us; error 24 us; skew 500 ppm
I20260928 02:51:43.149195 29325 webserver.cc:460] Webserver started at http://127.19.139.129:46461/ using document root <none> and password file <none>
I20260928 02:51:43.149752 29325 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:43.149818 29325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:43.149997 29325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:43.151989 29325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "009ffc67a3554d93bd560fbd0c8b97a7"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "d58a895321a52e572eb9954aaa4bd01c"
server_key_iv: "6fb6609e2884c5e69b002f5e28e525d0"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.152571 29325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "009ffc67a3554d93bd560fbd0c8b97a7"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "d58a895321a52e572eb9954aaa4bd01c"
server_key_iv: "6fb6609e2884c5e69b002f5e28e525d0"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.156100 29325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:51:43.158514 29339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.159646 29325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:43.159749 29325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "009ffc67a3554d93bd560fbd0c8b97a7"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "d58a895321a52e572eb9954aaa4bd01c"
server_key_iv: "6fb6609e2884c5e69b002f5e28e525d0"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.159840 29325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:43.201352 29325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:43.202071 29325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:43.202252 29325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:43.202883 29325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:43.203819 29325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:43.203866 29325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.203917 29325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:43.203945 29325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.212049 29325 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:38049
I20260928 02:51:43.212080 29452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:38049 every 8 connection(s)
I20260928 02:51:43.213088 29325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:43.221668 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.214781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:38663 (local address 127.19.139.190:37259)
0928 02:51:43.214900 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:43.214903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:43.215935 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 02:51:43.217304 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:43.217307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:43.217356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:43.217435 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:43.218306 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.218824 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.219973 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.220266 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.220342 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:43.221316 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:43.221319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:43.221321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:43.221331 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:43.221346 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:43.221352 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:43.221396 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:43.221495 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:51:43.221548 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:51:43.222435 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.215179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37259 (local address 127.19.139.129:38663)
0928 02:51:43.215789 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:43.215830 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:43.217115 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:43.217611 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:43.217619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:43.217702 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:43.218199 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:43.218211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.218967 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.218970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:43.219876 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:43.219885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.220392 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.220395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:43.220537 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:43.221096 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:43.221098 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:43.221140 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:43.221200 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:43.221402 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:43.221406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:43.221410 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:51:43.221486 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:51:43.221702 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260928 02:51:43.223100 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29325
I20260928 02:51:43.223205 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:43.223518 20014 external_mini_cluster.cc:1428] Setting key ffa0a3790b8f047d0493bf608061fa36
I20260928 02:51:43.223642 29453 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37259
I20260928 02:51:43.223860 29453 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:43.224404 29453 heartbeater.cc:511] Master 127.19.139.190:37259 requested a full tablet report, sending...
I20260928 02:51:43.225291 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37259
--builtin_ntp_servers=127.19.139.148:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:51:43.225842 29265 ts_manager.cc:194] Registered new tserver with Master: 009ffc67a3554d93bd560fbd0c8b97a7 (127.19.139.129:38049)
I20260928 02:51:43.226934 29265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.129:38663
W20260928 02:51:43.332442 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:43.332718 29456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:43.332758 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:43.336902 29456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:43.336969 29456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:43.337029 29456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:43.341995 29456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:43.343029 29456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:43.344475 29456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:51:43.353976 29456 server_base.cc:1034] running on GCE node
W20260928 02:51:43.353955 29464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:43.354139 29461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:43.353987 29462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:43.354552 29456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:43.355201 29456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:43.356349 29456 hybrid_clock.cc:648] HybridClock initialized: now 1790563903356330 us; error 32 us; skew 500 ppm
I20260928 02:51:43.358135 29456 webserver.cc:460] Webserver started at http://127.19.139.130:38751/ using document root <none> and password file <none>
I20260928 02:51:43.358697 29456 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:43.358763 29456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:43.358945 29456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:43.361081 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "f9759266760240829e14292a95fad1c4"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "7696782811e0f1b6de0ea1dff24f24a1"
server_key_iv: "cf6054c3438641cf8bc0e2982638f17a"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.361629 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "f9759266760240829e14292a95fad1c4"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "7696782811e0f1b6de0ea1dff24f24a1"
server_key_iv: "cf6054c3438641cf8bc0e2982638f17a"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.365392 29456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:51:43.367796 29470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.368993 29456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:43.369102 29456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "f9759266760240829e14292a95fad1c4"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "7696782811e0f1b6de0ea1dff24f24a1"
server_key_iv: "cf6054c3438641cf8bc0e2982638f17a"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.369196 29456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:43.392508 29456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:43.393221 29456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:43.393397 29456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:43.393982 29456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:43.394932 29456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:43.394977 29456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.395031 29456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:43.395058 29456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.403625 29456 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:37209
I20260928 02:51:43.403661 29583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:37209 every 8 connection(s)
I20260928 02:51:43.404603 29456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:43.411283 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29456
I20260928 02:51:43.411375 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:43.411652 20014 external_mini_cluster.cc:1428] Setting key 5cbc52023bcadb9cf4248bf5d8650e8b
I20260928 02:51:43.413609 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37259
--builtin_ntp_servers=127.19.139.148:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:51:43.414389 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.406599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:39595 (local address 127.19.139.190:37259)
0928 02:51:43.406735 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:43.406738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:43.408131 (+ 1393us) server_negotiation.cc:407] Connection header received
0928 02:51:43.409584 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:43.409587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:43.409644 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:43.409716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:43.410608 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.411145 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.412465 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.412700 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.412792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:43.413933 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:43.413936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:43.413938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:43.413949 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:43.413969 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:43.413984 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:43.414041 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:43.414184 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:51:43.414252 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:51:43.415699 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.407127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37259 (local address 127.19.139.130:39595)
0928 02:51:43.407973 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:43.408037 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:43.409370 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:43.409887 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:43.409894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:43.409981 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:43.410461 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:43.410484 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.411258 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.411262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:43.412364 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:43.412375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.412847 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.412852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:43.413045 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:43.413640 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:43.413642 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:43.413693 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:43.413781 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:43.414052 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:43.414061 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:43.414066 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:51:43.414178 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:51:43.414639 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260928 02:51:43.416797 29584 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37259
I20260928 02:51:43.417043 29584 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:43.417672 29584 heartbeater.cc:511] Master 127.19.139.190:37259 requested a full tablet report, sending...
I20260928 02:51:43.418860 29265 ts_manager.cc:194] Registered new tserver with Master: f9759266760240829e14292a95fad1c4 (127.19.139.130:37209)
I20260928 02:51:43.419410 29265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.130:39595
W20260928 02:51:43.524250 29587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:43.524533 29587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:43.524574 29587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:43.528734 29587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:43.528795 29587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:43.528854 29587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:43.533589 29587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:43.534641 29587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:43.536013 29587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:43.545323 29592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:43.545336 29595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:43.545396 29587 server_base.cc:1034] running on GCE node
W20260928 02:51:43.545341 29593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:43.546008 29587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:43.546608 29587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:43.547760 29587 hybrid_clock.cc:648] HybridClock initialized: now 1790563903547741 us; error 34 us; skew 500 ppm
I20260928 02:51:43.549640 29587 webserver.cc:460] Webserver started at http://127.19.139.131:46343/ using document root <none> and password file <none>
I20260928 02:51:43.550202 29587 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:43.550271 29587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:43.550454 29587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:43.552516 29587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "5e1a72b1a24944e59c9c3a174b52a0dd"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "c26e47b0b1eea0878814e2e00b4e6d3f"
server_key_iv: "fcbca55c0304642c6b77057b556409d3"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.553042 29587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "5e1a72b1a24944e59c9c3a174b52a0dd"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "c26e47b0b1eea0878814e2e00b4e6d3f"
server_key_iv: "fcbca55c0304642c6b77057b556409d3"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.556704 29587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:43.559293 29601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.560612 29587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:43.560721 29587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "5e1a72b1a24944e59c9c3a174b52a0dd"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "c26e47b0b1eea0878814e2e00b4e6d3f"
server_key_iv: "fcbca55c0304642c6b77057b556409d3"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.560824 29587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:43.587536 29587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:43.588274 29587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:43.588450 29587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:43.589030 29587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:43.589987 29587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:43.590042 29587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.590122 29587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:43.590147 29587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.598361 29587 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:44521
I20260928 02:51:43.598398 29714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:44521 every 8 connection(s)
I20260928 02:51:43.599273 29587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:43.599709 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29587
I20260928 02:51:43.599817 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:43.600133 20014 external_mini_cluster.cc:1428] Setting key e8446d9a9bc48aada23ec8ca21644715
I20260928 02:51:43.608543 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.601039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:49451 (local address 127.19.139.190:37259)
0928 02:51:43.601166 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:43.601169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:43.602523 (+ 1354us) server_negotiation.cc:407] Connection header received
0928 02:51:43.603902 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:43.603905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:43.603962 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:43.604032 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:43.604971 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.605507 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.606669 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.606898 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.606984 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:43.608024 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:43.608028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:43.608031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:43.608044 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:43.608066 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:43.608074 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:43.608131 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:43.608326 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:51:43.608392 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:51:43.609397 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.601626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37259 (local address 127.19.139.131:49451)
0928 02:51:43.602389 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:43.602431 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:43.603692 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:43.604233 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:43.604240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:43.604329 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:43.604831 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:43.604844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.605629 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.605632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:43.606545 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:43.606555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:43.607037 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:43.607042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:43.607213 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:43.607760 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:43.607762 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:43.607804 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:43.607870 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:43.608181 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:43.608191 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:43.608197 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:43.608314 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:51:43.608602 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260928 02:51:43.610457 29715 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37259
I20260928 02:51:43.610683 29715 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:43.611151 29715 heartbeater.cc:511] Master 127.19.139.190:37259 requested a full tablet report, sending...
I20260928 02:51:43.612092 29265 ts_manager.cc:194] Registered new tserver with Master: 5e1a72b1a24944e59c9c3a174b52a0dd (127.19.139.131:44521)
I20260928 02:51:43.612681 29265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.131:49451
I20260928 02:51:43.613926 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:51:43.616640 29304 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:43.615688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.19.139.190:37259)
0928 02:51:43.615799 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:43.615803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:43.615967 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:51:43.616076 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:43.616079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:43.616142 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:43.616236 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:43.616526 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48146: BlockingRecv error: recv got EOF from 127.0.0.1:48146 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:51:43.616719 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29325
I20260928 02:51:43.621642 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29456
I20260928 02:51:43.626080 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29587
I20260928 02:51:43.630263 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29234
2026-09-28T02:51:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (884 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:43Z Disabled control of system clock
I20260928 02:51:43.656400 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35285
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:51:43.769729 29726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:43.770032 29726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:43.770074 29726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:43.774281 29726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:43.774371 29726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:43.774395 29726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:43.774415 29726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:43.774436 29726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:43.779305 29726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:35285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:43.780413 29726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:43.781862 29726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:43.790082 29732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:43.790107 29734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:43.790099 29731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:43.790261 29726 server_base.cc:1034] running on GCE node
I20260928 02:51:43.790881 29726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:43.791934 29726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:43.793087 29726 hybrid_clock.cc:648] HybridClock initialized: now 1790563903793056 us; error 48 us; skew 500 ppm
I20260928 02:51:43.794853 29726 webserver.cc:460] Webserver started at http://127.19.139.190:40627/ using document root <none> and password file <none>
I20260928 02:51:43.795393 29726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:43.795460 29726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:43.795640 29726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:43.797736 29726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "d1e859534be34a5db225144a73ffdf75"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "30444e5403a5479bc653dbfd3f6a491b"
server_key_iv: "b924a534073602cee48f3d0605519ea9"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.798273 29726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "d1e859534be34a5db225144a73ffdf75"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "30444e5403a5479bc653dbfd3f6a491b"
server_key_iv: "b924a534073602cee48f3d0605519ea9"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.801898 29726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:51:43.804301 29740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:43.805521 29726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:51:43.805634 29726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "d1e859534be34a5db225144a73ffdf75"
format_stamp: "Formatted at 2026-09-28 02:51:43 on dist-test-slave-5vc0"
server_key: "30444e5403a5479bc653dbfd3f6a491b"
server_key_iv: "b924a534073602cee48f3d0605519ea9"
server_key_version: "encryptionkey@0"
I20260928 02:51:43.805737 29726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:43.810175 29726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:43.810758 29726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:43.810909 29726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:43.817219 29726 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:35285
I20260928 02:51:43.817224 29792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:35285 every 8 connection(s)
I20260928 02:51:43.818122 29726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:43.821360 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:43.821724 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29726
I20260928 02:51:43.821828 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:43.822131 20014 external_mini_cluster.cc:1428] Setting key 1a6e647e298f6db1ec79f1d715406331
I20260928 02:51:43.827602 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:43.823517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.19.139.190:35285)
0928 02:51:43.824196 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:43.824215 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:43.824251 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:51:43.824940 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:43.824958 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:43.825312 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:43.825645 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:43.826386 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:43.826391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:43.826402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:43.826413 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:43.826466 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:43.826482 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:43.826637 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:43.826706 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:51:43.826929 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260928 02:51:43.828245 29793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75: Bootstrap starting.
I20260928 02:51:43.830531 29793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:43.831228 29793 log.cc:824] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:43.833187 29793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75: No bootstrap required, opened a new log
I20260928 02:51:43.836829 29793 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e859534be34a5db225144a73ffdf75" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35285 } }
I20260928 02:51:43.837060 29793 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:43.837107 29793 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e859534be34a5db225144a73ffdf75, State: Initialized, Role: FOLLOWER
I20260928 02:51:43.837499 29793 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e859534be34a5db225144a73ffdf75" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35285 } }
I20260928 02:51:43.837625 29793 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:43.837671 29793 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:43.837740 29793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:43.838616 29793 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e859534be34a5db225144a73ffdf75" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35285 } }
I20260928 02:51:43.838913 29793 leader_election.cc:302] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1e859534be34a5db225144a73ffdf75; no voters:
I20260928 02:51:43.839196 29793 leader_election.cc:288] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:43.839303 29798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:43.839496 29798 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [term 1 LEADER]: Becoming Leader. State: Replica: d1e859534be34a5db225144a73ffdf75, State: Running, Role: LEADER
I20260928 02:51:43.839841 29798 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e859534be34a5db225144a73ffdf75" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35285 } }
I20260928 02:51:43.840407 29793 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:43.841614 29799 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1e859534be34a5db225144a73ffdf75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e859534be34a5db225144a73ffdf75" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35285 } } }
I20260928 02:51:43.841605 29800 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e859534be34a5db225144a73ffdf75. Latest consensus state: current_term: 1 leader_uuid: "d1e859534be34a5db225144a73ffdf75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e859534be34a5db225144a73ffdf75" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35285 } } }
I20260928 02:51:43.841740 29799 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:43.841740 29800 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:43.842693 29807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:43.845393 29807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:43.850817 29807 catalog_manager.cc:1347] Generated new cluster ID: 0d0c0accfd7542eaaff868627fe730ad
I20260928 02:51:43.850876 29807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:43.865434 29807 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:43.865769 29807 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:43.871977 29807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1e859534be34a5db225144a73ffdf75: Generated new TSK 0
I20260928 02:51:43.872824 29807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:51:43.884665 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35285
--builtin_ntp_servers=127.19.139.148:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:51:43.992851 29817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:43.993134 29817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:43.993187 29817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:43.997301 29817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:43.997370 29817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:43.997449 29817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:44.002298 29817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:44.003418 29817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:44.004930 29817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:44.014520 29822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:44.014513 29823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:44.014513 29825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:44.014685 29817 server_base.cc:1034] running on GCE node
I20260928 02:51:44.015118 29817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:44.015789 29817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:44.016943 29817 hybrid_clock.cc:648] HybridClock initialized: now 1790563904016923 us; error 33 us; skew 500 ppm
I20260928 02:51:44.018755 29817 webserver.cc:460] Webserver started at http://127.19.139.129:44065/ using document root <none> and password file <none>
I20260928 02:51:44.019357 29817 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:44.019428 29817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:44.019655 29817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:44.021772 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "b7ec2480160944cb8707df67935b538f"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "9096ae0c2e0d46d9185d9333371c4890"
server_key_iv: "d76be6782f98064e26a01df1100511ff"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.022306 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "b7ec2480160944cb8707df67935b538f"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "9096ae0c2e0d46d9185d9333371c4890"
server_key_iv: "d76be6782f98064e26a01df1100511ff"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.025909 29817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:51:44.028409 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.029627 29817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:44.029744 29817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "b7ec2480160944cb8707df67935b538f"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "9096ae0c2e0d46d9185d9333371c4890"
server_key_iv: "d76be6782f98064e26a01df1100511ff"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.029847 29817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:44.034808 29817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:44.035496 29817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:44.035657 29817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:44.036275 29817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:44.037209 29817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:44.037256 29817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.037307 29817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:44.037336 29817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.045719 29817 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:33649
I20260928 02:51:44.045740 29944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:33649 every 8 connection(s)
I20260928 02:51:44.046617 29817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:44.050273 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29817
I20260928 02:51:44.050364 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:44.050627 20014 external_mini_cluster.cc:1428] Setting key babc842604276cf33277b9191d3662ba
I20260928 02:51:44.052624 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:44.048334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:45651 (local address 127.19.139.190:35285)
0928 02:51:44.048530 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:44.048534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:44.049531 (+ 997us) server_negotiation.cc:407] Connection header received
0928 02:51:44.050863 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:44.050867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:44.050912 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:44.051005 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:44.052216 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:44.052220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:44.052223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:44.052235 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:44.052257 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:44.052265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:44.052334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:44.052486 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:51:44.052521 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:51:44.052613 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35285
--builtin_ntp_servers=127.19.139.148:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:51:44.053750 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:44.048747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35285 (local address 127.19.139.129:45651)
0928 02:51:44.049393 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:44.049433 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:44.050641 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:44.051202 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:44.051210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:44.051308 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:44.051872 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:44.051874 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:44.051929 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:44.052018 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:44.052342 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:44.052350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:44.052356 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:44.052531 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:51:44.052788 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260928 02:51:44.054960 29945 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35285
I20260928 02:51:44.055199 29945 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:44.055595 29945 heartbeater.cc:511] Master 127.19.139.190:35285 requested a full tablet report, sending...
I20260928 02:51:44.057281 29757 ts_manager.cc:194] Registered new tserver with Master: b7ec2480160944cb8707df67935b538f (127.19.139.129:33649)
W20260928 02:51:44.161777 29948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:44.162055 29948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:44.162094 29948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:44.166100 29948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:44.166163 29948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:44.166220 29948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:44.171046 29948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:44.172101 29948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:44.173499 29948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:44.182780 29954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:44.182798 29953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:44.182804 29956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:44.182927 29948 server_base.cc:1034] running on GCE node
I20260928 02:51:44.183468 29948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:44.184092 29948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:44.185277 29948 hybrid_clock.cc:648] HybridClock initialized: now 1790563904185258 us; error 31 us; skew 500 ppm
I20260928 02:51:44.187026 29948 webserver.cc:460] Webserver started at http://127.19.139.130:33887/ using document root <none> and password file <none>
I20260928 02:51:44.187582 29948 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:44.187651 29948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:44.187863 29948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:44.189913 29948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "413e13c979df4a42b2b6eb2eacaeb364"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "801f17659c88b792c7c9d40f36d1d2b6"
server_key_iv: "3f3343b0628ce400a416183b8b53d4cc"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.190462 29948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "413e13c979df4a42b2b6eb2eacaeb364"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "801f17659c88b792c7c9d40f36d1d2b6"
server_key_iv: "3f3343b0628ce400a416183b8b53d4cc"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.194031 29948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:51:44.196519 29962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.197675 29948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:44.197781 29948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "413e13c979df4a42b2b6eb2eacaeb364"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "801f17659c88b792c7c9d40f36d1d2b6"
server_key_iv: "3f3343b0628ce400a416183b8b53d4cc"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.197889 29948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:44.202280 29948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:44.202863 29948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:44.203016 29948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:44.203543 29948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:44.204475 29948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:44.204522 29948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.204574 29948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:44.204602 29948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.212883 29948 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:40013
I20260928 02:51:44.212904 30075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:40013 every 8 connection(s)
I20260928 02:51:44.213824 29948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:44.218801 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 29948
I20260928 02:51:44.218905 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:44.219195 20014 external_mini_cluster.cc:1428] Setting key aa353d4fb6a29db8ede3fe251cfbf89c
I20260928 02:51:44.219740 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:44.215513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:41319 (local address 127.19.139.190:35285)
0928 02:51:44.215647 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:44.215650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:44.216759 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 02:51:44.217955 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:44.217958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:44.217996 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:44.218068 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:44.219173 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:44.219177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:44.219180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:44.219193 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:44.219218 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:44.219229 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:44.219458 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:44.219585 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:51:44.219630 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:51:44.220789 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:44.215929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35285 (local address 127.19.139.130:41319)
0928 02:51:44.216612 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:44.216651 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:44.217776 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:44.218217 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:44.218223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:44.218314 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:44.218854 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:44.218856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:44.218917 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:44.219012 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:44.219440 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:44.219448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:44.219454 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:44.219642 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 02:51:44.219910 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260928 02:51:44.221133 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35285
--builtin_ntp_servers=127.19.139.148:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:51:44.221851 30076 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35285
I20260928 02:51:44.222103 30076 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:44.222551 30076 heartbeater.cc:511] Master 127.19.139.190:35285 requested a full tablet report, sending...
I20260928 02:51:44.223583 29757 ts_manager.cc:194] Registered new tserver with Master: 413e13c979df4a42b2b6eb2eacaeb364 (127.19.139.130:40013)
W20260928 02:51:44.329384 30079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:44.329669 30079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:44.329710 30079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:44.333778 30079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:44.333841 30079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:44.333900 30079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:44.338820 30079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:44.339879 30079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:44.341401 30079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:44.351011 30087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:44.351074 30084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:44.351078 30085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:44.351229 30079 server_base.cc:1034] running on GCE node
I20260928 02:51:44.351656 30079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:44.352361 30079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:44.353523 30079 hybrid_clock.cc:648] HybridClock initialized: now 1790563904353494 us; error 41 us; skew 500 ppm
I20260928 02:51:44.355378 30079 webserver.cc:460] Webserver started at http://127.19.139.131:41481/ using document root <none> and password file <none>
I20260928 02:51:44.355944 30079 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:44.356012 30079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:44.356261 30079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:44.358355 30079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "ff51eb701765488fa36f5d8a35074f9a"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "ce62942a0533b09b3ababbd1446dcff5"
server_key_iv: "32f466520aaa29d3aa5f4b17a33a1350"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.359005 30079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "ff51eb701765488fa36f5d8a35074f9a"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "ce62942a0533b09b3ababbd1446dcff5"
server_key_iv: "32f466520aaa29d3aa5f4b17a33a1350"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.362722 30079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:51:44.365334 30093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.366561 30079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:44.366688 30079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "ff51eb701765488fa36f5d8a35074f9a"
format_stamp: "Formatted at 2026-09-28 02:51:44 on dist-test-slave-5vc0"
server_key: "ce62942a0533b09b3ababbd1446dcff5"
server_key_iv: "32f466520aaa29d3aa5f4b17a33a1350"
server_key_version: "encryptionkey@0"
I20260928 02:51:44.366791 30079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:44.371206 30079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:44.371793 30079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:44.371941 30079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:44.372538 30079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:44.373426 30079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:44.373472 30079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.373520 30079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:44.373548 30079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:44.381776 30079 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:37401
I20260928 02:51:44.381798 30206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:37401 every 8 connection(s)
I20260928 02:51:44.382668 30079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:44.386965 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 30079
I20260928 02:51:44.387077 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:44.387363 20014 external_mini_cluster.cc:1428] Setting key e448be002f199ab1109091fb6e47e5df
I20260928 02:51:44.388574 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:44.384340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:59297 (local address 127.19.139.190:35285)
0928 02:51:44.384459 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:44.384462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:44.385505 (+ 1043us) server_negotiation.cc:407] Connection header received
0928 02:51:44.386785 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:44.386789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:44.386837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:44.386904 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:44.388081 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:44.388084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:44.388086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:51:44.388095 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:44.388136 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:51:44.388147 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:44.388253 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:44.388399 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:51:44.388445 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:51:44.389575 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:44.384746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35285 (local address 127.19.139.131:59297)
0928 02:51:44.385366 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:44.385404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:44.386551 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:44.387103 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:44.387113 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:44.387235 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:44.387821 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:51:44.387823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:44.387871 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:51:44.387941 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:44.388244 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:44.388251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:44.388257 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:51:44.388431 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:51:44.388673 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260928 02:51:44.390462 30207 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35285
I20260928 02:51:44.390667 30207 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:44.391014 30207 heartbeater.cc:511] Master 127.19.139.190:35285 requested a full tablet report, sending...
I20260928 02:51:44.391889 29757 ts_manager.cc:194] Registered new tserver with Master: ff51eb701765488fa36f5d8a35074f9a (127.19.139.131:37401)
I20260928 02:51:44.401278 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:51:44.403867 29796 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:51:44.403075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.19.139.190:35285)
0928 02:51:44.403202 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:44.403206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:44.403352 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:51:44.403441 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:44.403443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:44.403475 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:44.403535 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:44.403741 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38678: BlockingRecv error: recv got EOF from 127.0.0.1:38678 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:51:44.404109 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29817
I20260928 02:51:44.408581 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29948
I20260928 02:51:44.414783 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 30079
I20260928 02:51:44.418877 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 29726
2026-09-28T02:51:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (788 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:44 dist-test-slave-5vc0 krb5kdc[30217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:44 dist-test-slave-5vc0 krb5kdc[30217](info): set up 2 sockets
Sep 28 02:51:44 dist-test-slave-5vc0 krb5kdc[30217](info): commencing operation
krb5kdc: starting...
W20260928 02:51:45.372815 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:45 dist-test-slave-5vc0 krb5kdc[30217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:51:45.490777 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:34487
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:45.600008 30230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:45.600335 30230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:45.600392 30230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:45.604570 30230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:45.604650 30230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:45.604676 30230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:45.604697 30230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:45.604717 30230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:45.610112 30230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:45.611212 30230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:45.612303 30230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:45.620803 30235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:45.620872 30230 server_base.cc:1034] running on GCE node
W20260928 02:51:45.620863 30236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:45.620803 30238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:45.621569 30230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:45.622764 30230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:45.623955 30230 hybrid_clock.cc:648] HybridClock initialized: now 1790563905623936 us; error 38 us; skew 500 ppm
Sep 28 02:51:45 dist-test-slave-5vc0 krb5kdc[30217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:45.626770 30230 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:51:45.627703 30230 webserver.cc:460] Webserver started at http://127.19.139.190:40943/ using document root <none> and password file <none>
I20260928 02:51:45.628281 30230 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:45.628347 30230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:45.628547 30230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:45.630546 30230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "eea8cee8de4b4de7aa0a859821ddc733"
format_stamp: "Formatted at 2026-09-28 02:51:45 on dist-test-slave-5vc0"
server_key: "9049c43bcb26576eb12c85c46a59f2f4"
server_key_iv: "858dc6ec369bbedb400e338988d75bfb"
server_key_version: "encryptionkey@0"
I20260928 02:51:45.631083 30230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "eea8cee8de4b4de7aa0a859821ddc733"
format_stamp: "Formatted at 2026-09-28 02:51:45 on dist-test-slave-5vc0"
server_key: "9049c43bcb26576eb12c85c46a59f2f4"
server_key_iv: "858dc6ec369bbedb400e338988d75bfb"
server_key_version: "encryptionkey@0"
I20260928 02:51:45.634801 30230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:51:45.637228 30245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:45.638470 30230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:45.638579 30230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "eea8cee8de4b4de7aa0a859821ddc733"
format_stamp: "Formatted at 2026-09-28 02:51:45 on dist-test-slave-5vc0"
server_key: "9049c43bcb26576eb12c85c46a59f2f4"
server_key_iv: "858dc6ec369bbedb400e338988d75bfb"
server_key_version: "encryptionkey@0"
I20260928 02:51:45.638692 30230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:45.649154 30230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:45.652093 30230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:45.652284 30230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:45.659017 30230 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:34487
I20260928 02:51:45.659267 30297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:34487 every 8 connection(s)
I20260928 02:51:45.659963 30230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:45.662770 30298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:45.666930 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 30230
I20260928 02:51:45.667050 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:51:45.667310 20014 external_mini_cluster.cc:1428] Setting key ba63ee11e10c7d449b06afee4073d8de
I20260928 02:51:45.668262 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: Bootstrap starting.
I20260928 02:51:45.670639 30298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:45.671344 30298 log.cc:824] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:45.673440 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: No bootstrap required, opened a new log
Sep 28 02:51:45 dist-test-slave-5vc0 krb5kdc[30217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:45.677111 30298 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea8cee8de4b4de7aa0a859821ddc733" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34487 } }
I20260928 02:51:45.677333 30298 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:45.677371 30298 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea8cee8de4b4de7aa0a859821ddc733, State: Initialized, Role: FOLLOWER
I20260928 02:51:45.677863 30298 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea8cee8de4b4de7aa0a859821ddc733" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34487 } }
I20260928 02:51:45.678025 30298 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:45.678090 30298 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:45.678198 30298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:45.679283 30298 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea8cee8de4b4de7aa0a859821ddc733" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34487 } }
I20260928 02:51:45.679669 30298 leader_election.cc:302] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea8cee8de4b4de7aa0a859821ddc733; no voters:
I20260928 02:51:45.680011 30298 leader_election.cc:288] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:45.680106 30303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:45.680398 30303 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [term 1 LEADER]: Becoming Leader. State: Replica: eea8cee8de4b4de7aa0a859821ddc733, State: Running, Role: LEADER
I20260928 02:51:45.680964 30303 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea8cee8de4b4de7aa0a859821ddc733" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34487 } }
I20260928 02:51:45.681236 30298 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:45.682193 30303 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea8cee8de4b4de7aa0a859821ddc733. Latest consensus state: current_term: 1 leader_uuid: "eea8cee8de4b4de7aa0a859821ddc733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea8cee8de4b4de7aa0a859821ddc733" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34487 } } }
I20260928 02:51:45.682267 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:45.668728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.19.139.190:34487)
0928 02:51:45.669203 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:45.669219 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:45.669256 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:51:45.669972 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:45.669990 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:45.670333 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:45.670655 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:45.671433 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:45.672513 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:45.673521 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:45.673996 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:45.674142 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:45.676730 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:45.676735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:45.676749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:45.676775 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:45.678691 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:45.679100 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:45.679106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:45.679112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:45.679263 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:45.679527 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:45.679530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:45.679532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:45.680068 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:45.680271 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:45.680947 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 02:51:45.681270 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":140,"threads_started":1}
I20260928 02:51:45.682576 30305 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea8cee8de4b4de7aa0a859821ddc733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea8cee8de4b4de7aa0a859821ddc733" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 34487 } } }
I20260928 02:51:45.682674 30305 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:45.682835 30303 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733 [sys.catalog]: This master's current role is: LEADER
W20260928 02:51:45.686165 30318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:51:45.686244 30318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:51:45.686316 30315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:45.687148 30315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:45.693090 30315 catalog_manager.cc:1347] Generated new cluster ID: 509aab829eb54d189d46197c5f7bfce6
I20260928 02:51:45.693222 30315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:45.701412 30315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:45.702466 30315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:45.713428 30315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: Generated new TSK 0
I20260928 02:51:45.714184 30315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:51:45.751921 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:34487
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:45.885067 30324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:45.885411 30324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:45.885469 30324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:45.889465 30324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:45.889519 30324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:45.889577 30324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:51:45.894747 30324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:45.895762 30324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:45.896763 30324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:45.906013 30330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:45.906045 30329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:45.906145 30324 server_base.cc:1034] running on GCE node
W20260928 02:51:45.906018 30332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:45.906661 30324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:45.907296 30324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:45.908447 30324 hybrid_clock.cc:648] HybridClock initialized: now 1790563905908428 us; error 31 us; skew 500 ppm
Sep 28 02:51:45 dist-test-slave-5vc0 krb5kdc[30217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:45.911140 30324 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:51:45.912026 30324 webserver.cc:460] Webserver started at http://127.19.139.129:33825/ using document root <none> and password file <none>
I20260928 02:51:45.912612 30324 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:45.912681 30324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:45.912881 30324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:45.914906 30324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "ff16dfb382ea450d849777841fe46dc0"
format_stamp: "Formatted at 2026-09-28 02:51:45 on dist-test-slave-5vc0"
server_key: "ea61f4f66326d330f8c2619ff3c53ec4"
server_key_iv: "a325f332dad0d45d88bf551b52b55ffc"
server_key_version: "encryptionkey@0"
I20260928 02:51:45.915441 30324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "ff16dfb382ea450d849777841fe46dc0"
format_stamp: "Formatted at 2026-09-28 02:51:45 on dist-test-slave-5vc0"
server_key: "ea61f4f66326d330f8c2619ff3c53ec4"
server_key_iv: "a325f332dad0d45d88bf551b52b55ffc"
server_key_version: "encryptionkey@0"
I20260928 02:51:45.919039 30324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:51:45.921417 30339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:45.922600 30324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:51:45.922709 30324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "ff16dfb382ea450d849777841fe46dc0"
format_stamp: "Formatted at 2026-09-28 02:51:45 on dist-test-slave-5vc0"
server_key: "ea61f4f66326d330f8c2619ff3c53ec4"
server_key_iv: "a325f332dad0d45d88bf551b52b55ffc"
server_key_version: "encryptionkey@0"
I20260928 02:51:45.922809 30324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:45.945493 30324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:45.948928 30324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:45.949120 30324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:45.949738 30324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:45.950587 30324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:45.950635 30324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:45.950690 30324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:45.950717 30324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:45.959040 30324 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:46505
I20260928 02:51:45.959062 30452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:46505 every 8 connection(s)
I20260928 02:51:45.959949 30324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:51:45 dist-test-slave-5vc0 krb5kdc[30217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:45.969241 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 30324
I20260928 02:51:45.969352 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:45.969656 20014 external_mini_cluster.cc:1428] Setting key c04bdedc490cf91ad2e84bb5d9ef14ee
I20260928 02:51:45.972894 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:45.961759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:34431 (local address 127.19.139.190:34487)
0928 02:51:45.961898 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:45.961902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:45.962748 (+ 846us) server_negotiation.cc:407] Connection header received
0928 02:51:45.964087 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:45.964090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:45.964145 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:45.964236 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:45.965388 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:45.965920 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:45.967013 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:45.967238 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:45.967323 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:45.969745 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:45.969749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:45.969752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:45.969778 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:45.971007 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:45.971467 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:45.971469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:45.971471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:45.971543 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:45.971849 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:45.971852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:45.971854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:45.972025 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:45.972114 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:45.972614 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 02:51:45.972734 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:51:45.973694 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:45.962174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:34487 (local address 127.19.139.129:34431)
0928 02:51:45.962608 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:45.962649 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:45.963885 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:45.964411 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:45.964418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:45.964775 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:45.965251 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:45.965264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:45.966031 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:45.966034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:45.966902 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:45.966911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:45.967374 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:45.967379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:45.967539 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:45.968035 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:45.968051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:45.969544 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:45.971113 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:45.971118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:45.971133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:45.971376 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:45.971628 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:45.971631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:45.971633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:45.971767 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:45.972118 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:45.972125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:45.972393 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 02:51:45.972560 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 02:51:45.972883 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260928 02:51:45.974841 30453 heartbeater.cc:347] Connected to a master server at 127.19.139.190:34487
I20260928 02:51:45.975057 30453 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:45.975518 30453 heartbeater.cc:511] Master 127.19.139.190:34487 requested a full tablet report, sending...
I20260928 02:51:45.977149 30262 ts_manager.cc:194] Registered new tserver with Master: ff16dfb382ea450d849777841fe46dc0 (127.19.139.129:46505)
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:51:45.978569 30262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:34431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:51:45.995633 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:34487
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:46.106714 30458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:46.107009 30458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:46.107053 30458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:46.111241 30458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:46.111312 30458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:46.111375 30458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:51:46.117300 30458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:46.118404 30458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:46.119422 30458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:46.129379 30464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:46.129356 30466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:46.129355 30463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:46.129411 30458 server_base.cc:1034] running on GCE node
I20260928 02:51:46.130112 30458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:46.130745 30458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:46.131897 30458 hybrid_clock.cc:648] HybridClock initialized: now 1790563906131875 us; error 41 us; skew 500 ppm
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563906, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:46.134682 30458 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:51:46.135584 30458 webserver.cc:460] Webserver started at http://127.19.139.130:46589/ using document root <none> and password file <none>
I20260928 02:51:46.136148 30458 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:46.136260 30458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:46.136457 30458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:46.138513 30458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "00ac8df70664463f9bc0e1596f8ce1df"
format_stamp: "Formatted at 2026-09-28 02:51:46 on dist-test-slave-5vc0"
server_key: "aed10946f8cc6d3847ce5161a1b9d470"
server_key_iv: "7258cebe6e5abbe0db4eb5b3dbc67dbd"
server_key_version: "encryptionkey@0"
I20260928 02:51:46.139061 30458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "00ac8df70664463f9bc0e1596f8ce1df"
format_stamp: "Formatted at 2026-09-28 02:51:46 on dist-test-slave-5vc0"
server_key: "aed10946f8cc6d3847ce5161a1b9d470"
server_key_iv: "7258cebe6e5abbe0db4eb5b3dbc67dbd"
server_key_version: "encryptionkey@0"
I20260928 02:51:46.142772 30458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:51:46.145136 30473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:46.146282 30458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:46.146381 30458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "00ac8df70664463f9bc0e1596f8ce1df"
format_stamp: "Formatted at 2026-09-28 02:51:46 on dist-test-slave-5vc0"
server_key: "aed10946f8cc6d3847ce5161a1b9d470"
server_key_iv: "7258cebe6e5abbe0db4eb5b3dbc67dbd"
server_key_version: "encryptionkey@0"
I20260928 02:51:46.146468 30458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:46.180866 30458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:46.184291 30458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:46.184474 30458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:46.185050 30458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:46.185875 30458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:46.185922 30458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:46.185966 30458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:46.185993 30458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:46.194075 30458 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:43707
I20260928 02:51:46.194134 30586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:43707 every 8 connection(s)
I20260928 02:51:46.194988 30458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:46.202368 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 30458
I20260928 02:51:46.202471 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:46.202741 20014 external_mini_cluster.cc:1428] Setting key 84fb236cd2e647126de47b4b8b93fe5a
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563906, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:46.209018 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.196904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:44089 (local address 127.19.139.190:34487)
0928 02:51:46.197037 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.197040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.197941 (+ 901us) server_negotiation.cc:407] Connection header received
0928 02:51:46.199329 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.199333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.199391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.199470 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:46.200774 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.201318 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.202520 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.202766 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.202874 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.205380 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:46.205383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:46.205385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:46.205411 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:46.206913 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:46.207413 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:46.207417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:46.207419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:46.207512 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:46.207806 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:46.207809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:46.207811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:46.208050 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:46.208178 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.208699 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.208842 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:51:46.209816 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.197308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:34487 (local address 127.19.139.130:44089)
0928 02:51:46.197791 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.197834 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.199109 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.199664 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.199672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.200044 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:46.200616 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.200632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.201441 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.201445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.202362 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.202371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.202933 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.202937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.203105 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.203655 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:46.203672 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:46.205206 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:46.207029 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:46.207034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:46.207049 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:46.207324 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:46.207591 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:46.207593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:46.207594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:46.207727 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:46.208184 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:46.208191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:46.208449 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.208619 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.208894 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260928 02:51:46.210923 30587 heartbeater.cc:347] Connected to a master server at 127.19.139.190:34487
I20260928 02:51:46.211160 30587 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:46.211640 30587 heartbeater.cc:511] Master 127.19.139.190:34487 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:51:46.212646 30262 ts_manager.cc:194] Registered new tserver with Master: 00ac8df70664463f9bc0e1596f8ce1df (127.19.139.130:43707)
I20260928 02:51:46.213296 30262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:44089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:51:46.231758 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:34487
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:46.342736 30592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:46.343053 30592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:46.343116 30592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:46.347486 30592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:46.347565 30592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:46.347630 30592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:51:46.353276 30592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:51:46.354367 30592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:46.355388 30592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:46.365199 30600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:46.365263 30592 server_base.cc:1034] running on GCE node
W20260928 02:51:46.365238 30597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:46.365208 30598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:46.365969 30592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:46.366616 30592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:46.367784 30592 hybrid_clock.cc:648] HybridClock initialized: now 1790563906367750 us; error 45 us; skew 500 ppm
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563906, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:46.370609 30592 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:51:46.371464 30592 webserver.cc:460] Webserver started at http://127.19.139.131:42595/ using document root <none> and password file <none>
I20260928 02:51:46.371966 30592 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:46.372026 30592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:46.372228 30592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:46.374195 30592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "5407a8593d4341b48a4998eeac5433fd"
format_stamp: "Formatted at 2026-09-28 02:51:46 on dist-test-slave-5vc0"
server_key: "9300d3c51b364b96786b0587ceab3283"
server_key_iv: "6fd361a948fc0117c7cd5cc70cdf2760"
server_key_version: "encryptionkey@0"
I20260928 02:51:46.374701 30592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "5407a8593d4341b48a4998eeac5433fd"
format_stamp: "Formatted at 2026-09-28 02:51:46 on dist-test-slave-5vc0"
server_key: "9300d3c51b364b96786b0587ceab3283"
server_key_iv: "6fd361a948fc0117c7cd5cc70cdf2760"
server_key_version: "encryptionkey@0"
I20260928 02:51:46.378417 30592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:51:46.380860 30607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:46.382134 30592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:51:46.382232 30592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "5407a8593d4341b48a4998eeac5433fd"
format_stamp: "Formatted at 2026-09-28 02:51:46 on dist-test-slave-5vc0"
server_key: "9300d3c51b364b96786b0587ceab3283"
server_key_iv: "6fd361a948fc0117c7cd5cc70cdf2760"
server_key_version: "encryptionkey@0"
I20260928 02:51:46.382324 30592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:46.411077 30592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:46.413980 30592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:46.414131 30592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:46.414667 30592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:46.415591 30592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:46.415661 30592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:46.415716 30592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:46.415750 30592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 02:51:46.424788 30592 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46797
I20260928 02:51:46.424808 30720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46797 every 8 connection(s)
I20260928 02:51:46.425676 30592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:46.428179 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 30592
I20260928 02:51:46.428303 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:46.428612 20014 external_mini_cluster.cc:1428] Setting key b92af9ef311c61bc52412fade48118a9
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563906, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:51:46.439075 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.427430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:52567 (local address 127.19.139.190:34487)
0928 02:51:46.427566 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.427570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.428473 (+ 903us) server_negotiation.cc:407] Connection header received
0928 02:51:46.430061 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.430063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.430108 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.430179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:46.431510 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.432045 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.433210 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.433427 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.433520 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.435936 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:46.435939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:46.435941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:46.435967 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:46.437318 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:46.437772 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:46.437774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:46.437776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:46.437847 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:46.438132 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:46.438134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:46.438135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:46.438308 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:46.438377 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.438851 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.438950 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:51:46.439918 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.427801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:34487 (local address 127.19.139.131:52567)
0928 02:51:46.428302 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.428355 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.429841 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.430385 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.430395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.430884 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:46.431375 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.431390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.432140 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.432142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.433073 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.433082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.433564 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.433567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.433732 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.434249 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:46.434265 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:46.435782 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:46.437423 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:46.437427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:46.437440 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:46.437698 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:46.437913 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:46.437916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:46.437918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:46.438052 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:46.438377 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:46.438382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:46.438661 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.438819 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.439078 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260928 02:51:46.441087 30721 heartbeater.cc:347] Connected to a master server at 127.19.139.190:34487
I20260928 02:51:46.441334 30721 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:46.441917 30721 heartbeater.cc:511] Master 127.19.139.190:34487 requested a full tablet report, sending...
I20260928 02:51:46.442912 30262 ts_manager.cc:194] Registered new tserver with Master: 5407a8593d4341b48a4998eeac5433fd (127.19.139.131:46797)
I20260928 02:51:46.443496 30262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:52567
I20260928 02:51:46.454126 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:51:46.462440 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.455934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.19.139.190:34487)
0928 02:51:46.456054 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.456058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.456232 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:51:46.456349 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.456351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.456395 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.456460 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:46.457123 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.457621 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.458369 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.458574 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.458640 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.459791 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:46.459794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:46.459796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:46.459818 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:46.461164 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:46.461490 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:46.461492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:46.461493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:46.461559 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:46.461747 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:46.461748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:46.461750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:46.461949 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:46.462043 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.462228 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.462320 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:51:46.465077 30262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:51:46.467943 30262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:51:46.482712 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.476029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:43707 (local address 127.0.0.1:40272)
0928 02:51:46.477401 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.477413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.477551 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.478026 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.478030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.478057 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:46.478227 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.478234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.479549 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.479552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.480966 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.480976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.482068 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.482071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.482193 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.482236 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.482401 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.482539 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":108,"spinlock_wait_cycles":7680,"thread_start_us":64,"threads_started":1}
I20260928 02:51:46.482756 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.476202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.19.139.130:43707)
0928 02:51:46.476892 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.476904 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.477487 (+ 583us) server_negotiation.cc:407] Connection header received
0928 02:51:46.477681 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.477685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.477835 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.477975 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:46.478338 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.479419 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.481087 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.481967 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.482062 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.482166 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.482465 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.482573 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":147,"threads_started":1}
I20260928 02:51:46.483264 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.476114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46797 (local address 127.0.0.1:34728)
0928 02:51:46.477158 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.477209 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.477337 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.478026 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.478030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.478051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:46.478227 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.478234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.480655 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.480658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.481671 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.481682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.482880 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.482884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.482978 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.482999 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.483058 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.483128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":62,"spinlock_wait_cycles":14464,"thread_start_us":104,"threads_started":1}
I20260928 02:51:46.483703 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.476484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.19.139.131:46797)
0928 02:51:46.476963 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.476975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.477468 (+ 493us) server_negotiation.cc:407] Connection header received
0928 02:51:46.477536 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.477540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.477736 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.477878 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:46.479585 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.480530 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.481800 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.482571 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.482690 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.482813 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.483431 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.483534 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":132,"threads_started":1}
I20260928 02:51:46.483762 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.476786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:46505 (local address 127.0.0.1:60416)
0928 02:51:46.477201 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.477211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.477337 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.477875 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.477881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.477904 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:46.478039 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.478047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.479424 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.479426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.480977 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.480983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.483370 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.483374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.483459 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.483482 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.483545 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.483630 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":48,"spinlock_wait_cycles":3200,"thread_start_us":79,"threads_started":1}
I20260928 02:51:46.484098 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.476944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.19.139.129:46505)
0928 02:51:46.477274 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.477281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.477306 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:51:46.477523 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.477530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.477736 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.477878 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:46.478178 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.479320 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.482222 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.483059 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.483181 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.483312 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.483858 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.483957 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":66,"threads_started":1}
I20260928 02:51:46.486289 30521 tablet_service.cc:1467] Processing CreateTablet for tablet f4fae46d640b46ffa4c8392770943905 (DEFAULT_TABLE table=test-table [id=9b2c96104f1f4ea58fd2f6cab9313c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:46.486732 30387 tablet_service.cc:1467] Processing CreateTablet for tablet f4fae46d640b46ffa4c8392770943905 (DEFAULT_TABLE table=test-table [id=9b2c96104f1f4ea58fd2f6cab9313c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:46.486737 30655 tablet_service.cc:1467] Processing CreateTablet for tablet f4fae46d640b46ffa4c8392770943905 (DEFAULT_TABLE table=test-table [id=9b2c96104f1f4ea58fd2f6cab9313c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:51:46.487380 30521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4fae46d640b46ffa4c8392770943905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:46.487754 30655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4fae46d640b46ffa4c8392770943905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:46.487748 30387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4fae46d640b46ffa4c8392770943905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:46.493651 30738 tablet_bootstrap.cc:492] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: Bootstrap starting.
I20260928 02:51:46.495154 30740 tablet_bootstrap.cc:492] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: Bootstrap starting.
I20260928 02:51:46.496460 30739 tablet_bootstrap.cc:492] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Bootstrap starting.
I20260928 02:51:46.496836 30738 tablet_bootstrap.cc:654] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:46.497418 30740 tablet_bootstrap.cc:654] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:46.497766 30738 log.cc:824] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:46.498238 30740 log.cc:824] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:46.498804 30739 tablet_bootstrap.cc:654] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:46.499415 30739 log.cc:824] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:46.499825 30738 tablet_bootstrap.cc:492] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: No bootstrap required, opened a new log
I20260928 02:51:46.500020 30738 ts_tablet_manager.cc:1397] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:51:46.500898 30740 tablet_bootstrap.cc:492] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: No bootstrap required, opened a new log
I20260928 02:51:46.501000 30739 tablet_bootstrap.cc:492] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: No bootstrap required, opened a new log
I20260928 02:51:46.501087 30740 ts_tablet_manager.cc:1397] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:51:46.501180 30739 ts_tablet_manager.cc:1397] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:51:46.503973 30738 raft_consensus.cc:348] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.504316 30738 raft_consensus.cc:374] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:46.504376 30738 raft_consensus.cc:729] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ac8df70664463f9bc0e1596f8ce1df, State: Initialized, Role: FOLLOWER
I20260928 02:51:46.504377 30739 raft_consensus.cc:348] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.504616 30739 raft_consensus.cc:374] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:46.504659 30739 raft_consensus.cc:729] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5407a8593d4341b48a4998eeac5433fd, State: Initialized, Role: FOLLOWER
I20260928 02:51:46.504894 30738 consensus_queue.cc:260] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.505049 30739 consensus_queue.cc:260] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.505370 30740 raft_consensus.cc:348] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.505604 30740 raft_consensus.cc:374] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:46.505667 30740 raft_consensus.cc:729] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff16dfb382ea450d849777841fe46dc0, State: Initialized, Role: FOLLOWER
I20260928 02:51:46.505959 30738 ts_tablet_manager.cc:1428] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:51:46.505959 30739 ts_tablet_manager.cc:1428] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:51:46.506052 30740 consensus_queue.cc:260] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.506204 30721 heartbeater.cc:503] Master 127.19.139.190:34487 was elected leader, sending a full tablet report...
I20260928 02:51:46.506309 30587 heartbeater.cc:503] Master 127.19.139.190:34487 was elected leader, sending a full tablet report...
I20260928 02:51:46.506691 30453 heartbeater.cc:503] Master 127.19.139.190:34487 was elected leader, sending a full tablet report...
I20260928 02:51:46.506925 30740 ts_tablet_manager.cc:1428] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:51:46.607760 30746 raft_consensus.cc:482] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:51:46.607986 30746 raft_consensus.cc:504] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.609212 30746 leader_election.cc:288] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00ac8df70664463f9bc0e1596f8ce1df (127.19.139.130:43707), 5407a8593d4341b48a4998eeac5433fd (127.19.139.131:46797)
I20260928 02:51:46.613219 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.609575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46797 (local address 127.19.139.129:34541)
0928 02:51:46.609906 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.609917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.610043 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.610301 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.610303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.610329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:46.610447 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.610453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.611192 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.611195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.612130 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.612140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.612804 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.612807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.612915 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.612933 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.612990 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.613048 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":50,"spinlock_wait_cycles":12288,"thread_start_us":119,"threads_started":1}
I20260928 02:51:46.613236 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.609684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:34541 (local address 127.19.139.131:46797)
0928 02:51:46.609793 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.609796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.610014 (+ 218us) server_negotiation.cc:407] Connection header received
0928 02:51:46.610149 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.610153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.610206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.610278 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:46.610557 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.611078 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.612259 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.612683 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.612800 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.612858 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.613026 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.613101 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:51:46.613257 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.609558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:57853 (local address 127.19.139.130:43707)
0928 02:51:46.609709 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.609714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.609988 (+ 274us) server_negotiation.cc:407] Connection header received
0928 02:51:46.610164 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.610166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.610209 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.610272 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:51:46.610582 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.611063 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.612309 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.612695 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.612801 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.612853 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.613080 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.613156 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:51:46.613445 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.609447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:43707 (local address 127.19.139.129:57853)
0928 02:51:46.609866 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.609882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:46.610041 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:46.610302 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:46.610305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:46.610334 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:51:46.610466 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.610472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.611171 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.611176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.612181 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:46.612194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.612809 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.612813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:46.612953 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.612973 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:51:46.613056 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:51:46.613122 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":108,"spinlock_wait_cycles":28800,"thread_start_us":128,"threads_started":1}
I20260928 02:51:46.614009 30541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4fae46d640b46ffa4c8392770943905" candidate_uuid: "ff16dfb382ea450d849777841fe46dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00ac8df70664463f9bc0e1596f8ce1df" is_pre_election: true
I20260928 02:51:46.614019 30675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4fae46d640b46ffa4c8392770943905" candidate_uuid: "ff16dfb382ea450d849777841fe46dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5407a8593d4341b48a4998eeac5433fd" is_pre_election: true
I20260928 02:51:46.614315 30675 raft_consensus.cc:2454] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff16dfb382ea450d849777841fe46dc0 in term 0.
I20260928 02:51:46.614305 30541 raft_consensus.cc:2454] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff16dfb382ea450d849777841fe46dc0 in term 0.
I20260928 02:51:46.614832 30340 leader_election.cc:302] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ac8df70664463f9bc0e1596f8ce1df, ff16dfb382ea450d849777841fe46dc0; no voters:
I20260928 02:51:46.615129 30746 raft_consensus.cc:2781] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:51:46.615214 30746 raft_consensus.cc:482] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:51:46.615253 30746 raft_consensus.cc:3037] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:46.616348 30746 raft_consensus.cc:504] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.616679 30746 leader_election.cc:288] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [CANDIDATE]: Term 1 election: Requested vote from peers 00ac8df70664463f9bc0e1596f8ce1df (127.19.139.130:43707), 5407a8593d4341b48a4998eeac5433fd (127.19.139.131:46797)
I20260928 02:51:46.617029 30675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4fae46d640b46ffa4c8392770943905" candidate_uuid: "ff16dfb382ea450d849777841fe46dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5407a8593d4341b48a4998eeac5433fd"
I20260928 02:51:46.617029 30541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4fae46d640b46ffa4c8392770943905" candidate_uuid: "ff16dfb382ea450d849777841fe46dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00ac8df70664463f9bc0e1596f8ce1df"
I20260928 02:51:46.617161 30541 raft_consensus.cc:3037] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:46.617161 30675 raft_consensus.cc:3037] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:46.618443 30675 raft_consensus.cc:2454] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff16dfb382ea450d849777841fe46dc0 in term 1.
I20260928 02:51:46.618443 30541 raft_consensus.cc:2454] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff16dfb382ea450d849777841fe46dc0 in term 1.
I20260928 02:51:46.618750 30342 leader_election.cc:302] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5407a8593d4341b48a4998eeac5433fd, ff16dfb382ea450d849777841fe46dc0; no voters:
I20260928 02:51:46.618930 30746 raft_consensus.cc:2781] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:46.619146 30746 raft_consensus.cc:686] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 1 LEADER]: Becoming Leader. State: Replica: ff16dfb382ea450d849777841fe46dc0, State: Running, Role: LEADER
I20260928 02:51:46.619494 30746 consensus_queue.cc:237] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } }
I20260928 02:51:46.622563 30262 catalog_manager.cc:5495] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff16dfb382ea450d849777841fe46dc0 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "ff16dfb382ea450d849777841fe46dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff16dfb382ea450d849777841fe46dc0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 46505 } health_report { overall_health: HEALTHY } } }
I20260928 02:51:46.639557 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.635962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.19.139.129:46505)
0928 02:51:46.636074 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:46.636077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:46.636090 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:51:46.636230 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:46.636234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:46.636294 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:46.636387 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:51:46.636607 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.637109 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.637955 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:46.638270 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:46.638379 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:46.638731 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:51:46.639250 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:51:46.639343 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:46.639397 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:51:46.639448 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:51:46.645505 30541 raft_consensus.cc:1261] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 1 FOLLOWER]: Refusing update from remote peer ff16dfb382ea450d849777841fe46dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:46.645505 30675 raft_consensus.cc:1261] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 1 FOLLOWER]: Refusing update from remote peer ff16dfb382ea450d849777841fe46dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:51:46.646162 30746 consensus_queue.cc:1035] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ac8df70664463f9bc0e1596f8ce1df" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:46.646343 30749 consensus_queue.cc:1035] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5407a8593d4341b48a4998eeac5433fd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:51:46.653302 30754 mvcc.cc:205] Tried to move back new op lower bound from 7334149761613082624 to 7334149761516126208. Current Snapshot: MvccSnapshot[applied={T|T < 7334149761613082624}]
I20260928 02:51:46.654081 30752 mvcc.cc:205] Tried to move back new op lower bound from 7334149761613082624 to 7334149761516126208. Current Snapshot: MvccSnapshot[applied={T|T < 7334149761613082624}]
I20260928 02:51:46.662374 30262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:46.662539 30262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:51:46.664964 30262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eea8cee8de4b4de7aa0a859821ddc733: Sending DeleteTablet for 3 replicas of tablet f4fae46d640b46ffa4c8392770943905
I20260928 02:51:46.665704 30521 tablet_service.cc:1514] Processing DeleteTablet for tablet f4fae46d640b46ffa4c8392770943905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:46 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:40272
I20260928 02:51:46.665940 30387 tablet_service.cc:1514] Processing DeleteTablet for tablet f4fae46d640b46ffa4c8392770943905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:46 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:60416
I20260928 02:51:46.665951 30655 tablet_service.cc:1514] Processing DeleteTablet for tablet f4fae46d640b46ffa4c8392770943905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:51:46 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:34728
I20260928 02:51:46.666370 30762 tablet_replica.cc:330] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0: stopping tablet replica
I20260928 02:51:46.666843 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 30324
I20260928 02:51:46.666851 30762 raft_consensus.cc:2229] T f4fae46d640b46ffa4c8392770943905 P ff16dfb382ea450d849777841fe46dc0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:51:46.667505 30763 tablet_replica.cc:330] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: stopping tablet replica
I20260928 02:51:46.667883 30763 raft_consensus.cc:2229] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:46.668303 30763 raft_consensus.cc:2258] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:46.669737 30763 ts_tablet_manager.cc:1905] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:51:46.672279 30763 ts_tablet_manager.cc:1918] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:51:46.672371 30763 log.cc:1196] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/f4fae46d640b46ffa4c8392770943905
I20260928 02:51:46.672663 30763 ts_tablet_manager.cc:1939] T f4fae46d640b46ffa4c8392770943905 P 5407a8593d4341b48a4998eeac5433fd: Deleting consensus metadata
I20260928 02:51:46.674047 30248 catalog_manager.cc:4841] TS 5407a8593d4341b48a4998eeac5433fd (127.19.139.131:46797): tablet f4fae46d640b46ffa4c8392770943905 (table test-table [id=9b2c96104f1f4ea58fd2f6cab9313c6c]) successfully deleted
I20260928 02:51:46.675829 30761 tablet_replica.cc:330] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: stopping tablet replica
I20260928 02:51:46.676177 30761 raft_consensus.cc:2229] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:51:46.676515 30761 raft_consensus.cc:2258] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:51:46.678056 30761 ts_tablet_manager.cc:1905] T f4fae46d640b46ffa4c8392770943905 P 00ac8df70664463f9bc0e1596f8ce1df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:51:46.678349 30248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:46505 (error 108)
I20260928 02:51:46.679186 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 30458
I20260928 02:51:46.679396 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.679077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:46505 (local address 127.0.0.1:60446)
0928 02:51:46.679245 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.679315 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:46505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:51:46.680539 30248 catalog_manager.cc:4584] TS ff16dfb382ea450d849777841fe46dc0 (127.19.139.129:46505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4fae46d640b46ffa4c8392770943905: Network error: Client connection negotiation failed: client connection to 127.19.139.129:46505: connect: Connection refused (error 111)
I20260928 02:51:46.683970 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 30592
I20260928 02:51:46.688654 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 30230
I20260928 02:51:46.688685 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:46.685095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:43707 (local address 127.0.0.1:40288)
0928 02:51:46.688546 (+ 3451us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:46.688596 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3364}
2026-09-28T02:51:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2271 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30767](info): set up 2 sockets
Sep 28 02:51:46 dist-test-slave-5vc0 krb5kdc[30767](info): commencing operation
krb5kdc: starting...
W20260928 02:51:47.652966 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:51:47 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/rangeradmin_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/rangeradmin_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/rangerlookup_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/rangerlookup_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
I20260928 02:51:47.823056 20014 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAxMa1R/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/postgres -l logfile start
2026-09-28 02:51:50.229 UTC [30797] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:51:50.229 UTC [30797] LOG: listening on IPv4 address "127.19.139.148", port 45361
2026-09-28 02:51:50.232 UTC [30797] LOG: listening on Unix socket "/tmp/.s.PGSQL.45361"
2026-09-28 02:51:50.259 UTC [30800] LOG: database system was shut down at 2026-09-28 02:51:48 UTC
2026-09-28 02:51:50.263 UTC [30797] LOG: database system is ready to accept connections
I20260928 02:51:51.136721 20014 mini_postgres.cc:96] Postgres bound to 45361
2026-09-28 02:51:51.142 UTC [30808] FATAL: database "slave" does not exist
127.19.139.148:45361 - accepting connections
I20260928 02:51:51.143333 20014 mini_ranger.cc:161] Starting Ranger...
I20260928 02:51:51.158003 20014 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:51:51.625873 20014 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:51:51.625952 20014 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-admin
2026-09-28 02:51:51,890 [I] DB FLAVOR :POSTGRES
2026-09-28 02:51:51,890 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:51:51,890 [I] Checking connection..
2026-09-28 02:51:51,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:51:52,126 [I] Checking connection passed.
2026-09-28 02:51:52,126 [I] --------- Verifying version history table ---------
2026-09-28 02:51:52,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:51:52,386 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:51:52,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:51:52,656 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:51:52,656 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:51:52,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:51:52.891 UTC [30872] WARNING: there is no transaction in progress
2026-09-28 02:51:52,896 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:51:52,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:51:53,141 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:51:53,141 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:51:53,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:51:53,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:51:53,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5vc0.c.gcp-upstream.internal','N') ;"
2026-09-28 02:51:53,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:51:54,093 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:51:54,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:51:54,327 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:51:54,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:51:54,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:51:54,820 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:51:54,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:51:55.782 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.799 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.816 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.833 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.850 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.869 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.936 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.942 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.956 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.967 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.977 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:55.991 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:56.001 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:56.007 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:56.015 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:56.023 UTC [30980] WARNING: there is no transaction in progress
2026-09-28 02:51:56,673 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:51:56,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:51:56,919 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:51:56,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:51:57,174 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:51:57,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:51:57,407 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:51:57,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:51:57,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:51:57,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5vc0.c.gcp-upstream.internal';"
2026-09-28 02:51:58,157 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:51:58,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:51:58,393 [I] DB_PATCHES have already been applied
I20260928 02:51:58.398801 20014 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAxMa1R/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:51:58.398867 20014 mini_ranger.cc:193] Using host: 127.19.139.148
I20260928 02:51:58.402130 20014 mini_ranger.cc:227] Ranger admin URL: http://127.19.139.148:45733
Sep 28, 2026 2:51:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:51:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:51:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:51:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:51:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:51:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:51:59.390759 20014 mini_ranger.cc:160] Time spent starting Ranger: real 8.247s user 0.003s sys 0.006s
Sep 28, 2026 2:52:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:52:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:52:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:52:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:52:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:52:21.178181 20014 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/rangerkms_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/rangerkms_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.139.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.139.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:52:21 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:52:21.265200 20014 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:52:21.274967 20014 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:52:22.165604 20014 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:52:22.165696 20014 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms
I20260928 02:52:22.165803 20014 mini_ranger_kms.cc:227] Using postgres at 127.19.139.148:45361
2026-09-28 02:52:22,321 [I] DB FLAVOR :POSTGRES
2026-09-28 02:52:22,322 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:52:22,322 [I] Checking connection
2026-09-28 02:52:22,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:52:22,559 [I] connection success
2026-09-28 02:52:22,559 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:52:22,559 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:52:22,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:52:22,814 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:52:22,815 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:52:22,815 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:52:22,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:45361/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:52:23,098 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:52:23.574282 20014 mini_ranger_kms.cc:310] Created kms service
I20260928 02:52:23.754004 20014 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:52:23.795703 20014 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:52:23 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:52:24.170189 20014 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:52:24.170356 20014 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:52:24.170377 20014 mini_ranger_kms.cc:240] Using host: 127.19.139.148
I20260928 02:52:24.173928 20014 mini_ranger_kms.cc:276] Ranger KMS PID: 31160
I20260928 02:52:24.174094 20014 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.139.148:39981
Sep 28, 2026 2:52:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:52:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:52:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:52:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:52:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:52:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:52:26.243860 20014 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.979s user 0.018s sys 0.122s
I20260928 02:52:26.243990 20014 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:52:26.244094 20014 mini_ranger_kms.cc:401] 127.19.139.148:39981/kms/v1/keys
Sep 28 02:52:26 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563946, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.139.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:52:26 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563946, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.139.148@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
Sep 28 02:52:27 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563946, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.139.148@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
Sep 28 02:52:29 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:52:29.315435 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40979
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:40979
--ranger_config_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:39981/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:29.428123 31211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:29.428467 31211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:29.428509 31211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:29.432658 31211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:29.432739 31211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:29.432762 31211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:29.432780 31211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:29.432801 31211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:29.438577 31211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.139.148:39981/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:40979
--ranger_config_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:52:29.439795 31211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:29.440912 31211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:29.449350 31216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:29.449397 31211 server_base.cc:1034] running on GCE node
W20260928 02:52:29.449589 31219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:29.449362 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:29.450212 31211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:29.451323 31211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:29.452486 31211 hybrid_clock.cc:648] HybridClock initialized: now 1790563949452462 us; error 46 us; skew 500 ppm
Sep 28 02:52:29 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:29.459149 31211 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:52:29.460228 31211 webserver.cc:460] Webserver started at http://127.19.139.190:35155/ using document root <none> and password file <none>
I20260928 02:52:29.460816 31211 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:29.460925 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:29.461159 31211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:52:29 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:52:29.509441 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "8d3d12c54d0647028b68a0f28b60d84a"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-5vc0"
server_key: "055875682f3ff7f7ce2fc210dc07c76d"
server_key_iv: "94efa385b8a0a440f13219034502ff05"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:29.510183 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "8d3d12c54d0647028b68a0f28b60d84a"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-5vc0"
server_key: "055875682f3ff7f7ce2fc210dc07c76d"
server_key_iv: "94efa385b8a0a440f13219034502ff05"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:29.515076 31211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:52:29.529517 31226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.530941 31211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:29.531083 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "8d3d12c54d0647028b68a0f28b60d84a"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-5vc0"
server_key: "055875682f3ff7f7ce2fc210dc07c76d"
server_key_iv: "94efa385b8a0a440f13219034502ff05"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:29.531211 31211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:29.553490 31211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:29.556408 31211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:29.556586 31211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:29.563903 31211 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:40979
I20260928 02:52:29.563928 31278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:40979 every 8 connection(s)
I20260928 02:52:29.564917 31211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:29.568454 31279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:29.573087 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 31211
I20260928 02:52:29.573223 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:52:29.573513 20014 external_mini_cluster.cc:1428] Setting key 2f725f420515dddde405e83af62ded47
I20260928 02:52:29.575690 31279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a: Bootstrap starting.
I20260928 02:52:29.578503 31279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:29.579365 31279 log.cc:824] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:29.581581 31279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a: No bootstrap required, opened a new log
Sep 28 02:52:29 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:52:29.586115 31279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d12c54d0647028b68a0f28b60d84a" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40979 } }
I20260928 02:52:29.586393 31279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:29.586479 31279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3d12c54d0647028b68a0f28b60d84a, State: Initialized, Role: FOLLOWER
I20260928 02:52:29.586985 31279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d12c54d0647028b68a0f28b60d84a" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40979 } }
I20260928 02:52:29.587141 31279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:29.587172 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.575052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.19.139.190:40979)
0928 02:52:29.575473 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.575492 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.575528 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:52:29.576242 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.576262 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.576636 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.576950 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.577759 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.578805 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.579688 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.580363 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.580516 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.582864 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.582870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.582884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.582910 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.584422 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.584828 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.584833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.584837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.584941 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.585205 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.585208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.585209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.585593 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.585779 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.586055 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.586342 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260928 02:52:29.587222 31279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:29.587420 31279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:29.588559 31279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d12c54d0647028b68a0f28b60d84a" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40979 } }
I20260928 02:52:29.588922 31279 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3d12c54d0647028b68a0f28b60d84a; no voters:
I20260928 02:52:29.589254 31279 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:29.589371 31284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:29.589639 31284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [term 1 LEADER]: Becoming Leader. State: Replica: 8d3d12c54d0647028b68a0f28b60d84a, State: Running, Role: LEADER
I20260928 02:52:29.589977 31284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d12c54d0647028b68a0f28b60d84a" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40979 } }
I20260928 02:52:29.590566 31279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:29.591048 31284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3d12c54d0647028b68a0f28b60d84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d12c54d0647028b68a0f28b60d84a" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40979 } } }
I20260928 02:52:29.591238 31284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:29.591466 31284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3d12c54d0647028b68a0f28b60d84a. Latest consensus state: current_term: 1 leader_uuid: "8d3d12c54d0647028b68a0f28b60d84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3d12c54d0647028b68a0f28b60d84a" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40979 } } }
I20260928 02:52:29.591594 31284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:29.591879 31291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:29.595541 31291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:29.595928 31279 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:52:29.606479 31291 catalog_manager.cc:1347] Generated new cluster ID: 464d16aa1b23448496319b9c5fa21384
I20260928 02:52:29.606628 31291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:29.625214 31291 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:29.626277 31291 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:29.632997 31291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a: Generated new TSK 0
I20260928 02:52:29.634089 31291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:52:30.335631 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:39981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40979
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:30.486658 31324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:30.486928 31324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:30.486989 31324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:30.491245 31324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:30.491313 31324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:30.491394 31324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:52:30.498682 31324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.139.148:39981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:52:30.500097 31324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:30.501420 31324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:30.513985 31332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:30.513958 31324 server_base.cc:1034] running on GCE node
W20260928 02:52:30.513945 31331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:30.513914 31334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:30.514678 31324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:30.515319 31324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:30.516484 31324 hybrid_clock.cc:648] HybridClock initialized: now 1790563950516448 us; error 54 us; skew 500 ppm
Sep 28 02:52:30 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:30.519376 31324 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:52:30.520327 31324 webserver.cc:460] Webserver started at http://127.19.139.129:41181/ using document root <none> and password file <none>
I20260928 02:52:30.520951 31324 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:30.521019 31324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:30.521236 31324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:52:30 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:52:30.537271 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "93159a5963a44240830d0b98251ae012"
format_stamp: "Formatted at 2026-09-28 02:52:30 on dist-test-slave-5vc0"
server_key: "dbe39fdcd434e57effe0200735d41dd6"
server_key_iv: "a8e5ab4278069ddbcf383231da210533"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:30.539464 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "93159a5963a44240830d0b98251ae012"
format_stamp: "Formatted at 2026-09-28 02:52:30 on dist-test-slave-5vc0"
server_key: "dbe39fdcd434e57effe0200735d41dd6"
server_key_iv: "a8e5ab4278069ddbcf383231da210533"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:30.546200 31324 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260928 02:52:30.572618 31341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:30.586004 31324 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.002s sys 0.000s
I20260928 02:52:30.586166 31324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "93159a5963a44240830d0b98251ae012"
format_stamp: "Formatted at 2026-09-28 02:52:30 on dist-test-slave-5vc0"
server_key: "dbe39fdcd434e57effe0200735d41dd6"
server_key_iv: "a8e5ab4278069ddbcf383231da210533"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:30.586325 31324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:30.616070 31324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:30.619901 31324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:30.620251 31324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:30.621038 31324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:30.622135 31324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:30.622243 31324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:30.622359 31324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:30.622435 31324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:30.634750 31324 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:37817
I20260928 02:52:30.637053 31324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:30.641111 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 31324
I20260928 02:52:30.641242 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:30.641553 20014 external_mini_cluster.cc:1428] Setting key f1c9b5f6fe1ecf54d5ca0a2d1ffe37fc
I20260928 02:52:30.639993 31456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:37817 every 8 connection(s)
Sep 28 02:52:30 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:52:30.662412 31459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:30.642788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40979 (local address 127.19.139.129:49433)
0928 02:52:30.644350 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:30.644405 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:30.645408 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:30.646323 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:30.646332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:30.646666 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:30.647328 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:30.647348 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:30.648490 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:30.648493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:30.649865 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:30.649876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:30.650641 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:30.650644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:30.650840 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:30.651535 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:30.651556 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:30.653305 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:30.658598 (+ 5293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:30.658605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:30.658626 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:30.659031 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:30.659419 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:30.659422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:30.659424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:30.659665 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:30.660251 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:30.660259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:30.660620 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 02:52:30.660862 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 02:52:30.661258 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":150,"threads_started":1}
I20260928 02:52:30.663213 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:30.643844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:49433 (local address 127.19.139.190:40979)
0928 02:52:30.645827 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:30.645831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:30.645844 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:52:30.645883 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:30.645886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:30.645930 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:30.646008 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:30.647491 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:30.648352 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:30.650011 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:30.650380 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:30.650471 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:30.656458 (+ 5987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:30.656462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:30.656465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:30.656499 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:30.658426 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:30.659178 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:30.659181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:30.659183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:30.659292 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:30.659803 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:30.659806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:30.659808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:30.660054 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:30.660193 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:30.662897 (+ 2704us) server_negotiation.cc:299] Negotiation successful
0928 02:52:30.663039 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1905,"thread_start_us":74,"threads_started":1}
I20260928 02:52:30.664572 31457 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40979
I20260928 02:52:30.664848 31457 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:30.665449 31457 heartbeater.cc:511] Master 127.19.139.190:40979 requested a full tablet report, sending...
I20260928 02:52:30.667253 31243 ts_manager.cc:194] Registered new tserver with Master: 93159a5963a44240830d0b98251ae012 (127.19.139.129:37817)
I20260928 02:52:30.668424 31243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:49433
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Sep 28 02:52:30 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:52:30.689385 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:39981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40979
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:30.847652 31463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:30.848821 31463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:30.848935 31463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:30.855113 31463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:30.855309 31463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:30.855445 31463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:52:30.863515 31463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.139.148:39981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:52:30.865365 31463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:30.866796 31463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:30.882436 31470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:30.882666 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:30.882432 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:30.883220 31463 server_base.cc:1034] running on GCE node
I20260928 02:52:30.883728 31463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:30.884620 31463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:30.885792 31463 hybrid_clock.cc:648] HybridClock initialized: now 1790563950885726 us; error 83 us; skew 500 ppm
Sep 28 02:52:30 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:30.889465 31463 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:52:30.890651 31463 webserver.cc:460] Webserver started at http://127.19.139.130:42297/ using document root <none> and password file <none>
I20260928 02:52:30.891494 31463 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:30.891626 31463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:30.891980 31463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:52:30 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:52:30.907800 31463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "295c0a2ce79f493cab0f7793950d0c1c"
format_stamp: "Formatted at 2026-09-28 02:52:30 on dist-test-slave-5vc0"
server_key: "71b2ba55b7d87f5ff33a0bc913e77984"
server_key_iv: "cb3dc9a2de85ccb6b3e88c55ac7a4e25"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:30.913429 31463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "295c0a2ce79f493cab0f7793950d0c1c"
format_stamp: "Formatted at 2026-09-28 02:52:30 on dist-test-slave-5vc0"
server_key: "71b2ba55b7d87f5ff33a0bc913e77984"
server_key_iv: "cb3dc9a2de85ccb6b3e88c55ac7a4e25"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:30.925828 31463 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260928 02:52:30.948243 31479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:30.950026 31463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:52:30.950168 31463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "295c0a2ce79f493cab0f7793950d0c1c"
format_stamp: "Formatted at 2026-09-28 02:52:30 on dist-test-slave-5vc0"
server_key: "71b2ba55b7d87f5ff33a0bc913e77984"
server_key_iv: "cb3dc9a2de85ccb6b3e88c55ac7a4e25"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:30.950307 31463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:30.968796 31463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:30.972707 31463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:30.972973 31463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:30.973762 31463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:30.974946 31463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:30.975052 31463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:30.975157 31463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:30.975219 31463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:30.986315 31463 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:44553
I20260928 02:52:30.987605 31463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:30.988864 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 31463
I20260928 02:52:30.988996 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:30.989283 20014 external_mini_cluster.cc:1428] Setting key 5b98907f9df25575d91021e339cd53ae
I20260928 02:52:31.000634 31592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:44553 every 8 connection(s)
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Sep 28 02:52:31 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:52:31.021432 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.007004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:54577 (local address 127.19.139.190:40979)
0928 02:52:31.007143 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.007147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.008396 (+ 1249us) server_negotiation.cc:407] Connection header received
0928 02:52:31.009603 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.009607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.009661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.009721 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:31.011273 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.012188 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.013803 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.014130 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.014199 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.017128 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:31.017132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:31.017134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:31.017161 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:31.018846 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.019598 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.019602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.019604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.019698 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.020113 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.020116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.020118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.020375 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:31.020442 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.021141 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.021276 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:52:31.023242 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.007555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40979 (local address 127.19.139.130:54577)
0928 02:52:31.008145 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.008288 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.009354 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.010121 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.010130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.010460 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:31.011111 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.011129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.012334 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.012337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.013663 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.013673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.014411 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.014414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.014606 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.015260 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:31.015280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:31.016943 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:31.018973 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:31.018983 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:31.019008 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:31.019395 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:31.019811 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:31.019814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:31.019816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:31.020008 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:31.020519 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:31.020526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:31.020868 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.021824 (+ 956us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.022186 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":154,"threads_started":1}
I20260928 02:52:31.024858 31594 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40979
I20260928 02:52:31.025102 31594 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:31.025686 31594 heartbeater.cc:511] Master 127.19.139.190:40979 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:52:31.026916 31243 ts_manager.cc:194] Registered new tserver with Master: 295c0a2ce79f493cab0f7793950d0c1c (127.19.139.130:44553)
I20260928 02:52:31.027662 31243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:54577
I20260928 02:52:31.028739 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:39981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40979
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:52:31 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
W20260928 02:52:31.203982 31598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:31.204455 31598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:31.204556 31598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:31.210577 31598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:31.210666 31598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:31.210752 31598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:52:31.218957 31598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.139.148:39981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:52:31.220927 31598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:31.223097 31598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:31.238121 31606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:31.239008 31604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:31.239601 31603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:31.241010 31598 server_base.cc:1034] running on GCE node
I20260928 02:52:31.241937 31598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:31.242992 31598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:31.260251 31598 hybrid_clock.cc:648] HybridClock initialized: now 1790563951260152 us; error 111 us; skew 500 ppm
Sep 28 02:52:31 dist-test-slave-5vc0 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563951, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:31.264405 31598 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:52:31.265791 31598 webserver.cc:460] Webserver started at http://127.19.139.131:40633/ using document root <none> and password file <none>
I20260928 02:52:31.266561 31598 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:31.266667 31598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:31.266953 31598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:52:31 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563951, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:52:31.289760 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "f01777e4680144bdaaa63b845b8778bd"
format_stamp: "Formatted at 2026-09-28 02:52:31 on dist-test-slave-5vc0"
server_key: "37894a755367480633890de9003db1d8"
server_key_iv: "bf7e7276bb916a214bcfcf7e822d96a7"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:31.291865 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "f01777e4680144bdaaa63b845b8778bd"
format_stamp: "Formatted at 2026-09-28 02:52:31 on dist-test-slave-5vc0"
server_key: "37894a755367480633890de9003db1d8"
server_key_iv: "bf7e7276bb916a214bcfcf7e822d96a7"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:31.297224 31598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:52:31.324662 31613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:31.326613 31598 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.002s
I20260928 02:52:31.326850 31598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "f01777e4680144bdaaa63b845b8778bd"
format_stamp: "Formatted at 2026-09-28 02:52:31 on dist-test-slave-5vc0"
server_key: "37894a755367480633890de9003db1d8"
server_key_iv: "bf7e7276bb916a214bcfcf7e822d96a7"
server_key_version: "kuduclusterkey@0"
I20260928 02:52:31.327077 31598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:31.349973 31598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:31.353701 31598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:31.353911 31598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:31.354682 31598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:31.355875 31598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:31.355948 31598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:31.356067 31598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:31.356105 31598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:31.369206 31598 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:33845
I20260928 02:52:31.370481 31598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:31.371179 31726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:33845 every 8 connection(s)
I20260928 02:52:31.380124 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 31598
I20260928 02:52:31.380294 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:31.380573 20014 external_mini_cluster.cc:1428] Setting key 1da3605f794d622c19a327c32a179bf2
Sep 28 02:52:31 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563951, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:52:31.387985 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.374442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:41611 (local address 127.19.139.190:40979)
0928 02:52:31.374590 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.374595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.374616 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:52:31.375158 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.375162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.375223 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.375327 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:31.376754 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.377591 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.379258 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.379640 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.379769 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.383655 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:31.383660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:31.383663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:31.383696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:31.385496 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.386147 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.386151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.386153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.386273 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.386685 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.386689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.386691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.386949 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:31.387067 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.387662 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.387801 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:52:31.389000 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.373215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40979 (local address 127.19.139.131:41611)
0928 02:52:31.373826 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.373884 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.374893 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.375521 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.375530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.375886 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:31.376582 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.376609 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.377715 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.377719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.379113 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.379125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.379807 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.379810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.381122 (+ 1312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.381821 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:31.381843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:31.383463 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:31.385655 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:31.385661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:31.385680 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:31.386035 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:31.386378 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:31.386381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:31.386383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:31.386569 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:31.387057 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:31.387063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:31.387386 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.387602 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.387934 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":160,"threads_started":1}
I20260928 02:52:31.390247 31727 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40979
I20260928 02:52:31.390516 31727 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:31.391149 31727 heartbeater.cc:511] Master 127.19.139.190:40979 requested a full tablet report, sending...
I20260928 02:52:31.392427 31243 ts_manager.cc:194] Registered new tserver with Master: f01777e4680144bdaaa63b845b8778bd (127.19.139.131:33845)
I20260928 02:52:31.393244 31243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:41611
I20260928 02:52:31.397318 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:52:31.409795 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.399808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.19.139.190:40979)
0928 02:52:31.399920 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.399924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.400197 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:52:31.400401 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.400404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.400453 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.400514 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:31.401561 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.402360 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.403544 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.403882 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.403964 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.405934 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:31.405938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:31.405941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:31.405969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:31.407697 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.408252 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.408255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.408257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.408376 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.408756 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.408760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.408762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.408990 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:31.409101 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.409451 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.409613 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:52:31.413606 31243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:31.417918 31243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:31.437884 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.428556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:33845 (local address 127.0.0.1:54722)
0928 02:52:31.430602 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.430654 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.430781 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.431492 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.431499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.431522 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:31.431655 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.431662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.433910 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.433914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.436089 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.436098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.437374 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.437377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.437488 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.437523 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.437618 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.437734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"thread_start_us":128,"threads_started":1}
I20260928 02:52:31.438549 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.429450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.19.139.131:33845)
0928 02:52:31.430351 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.430362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.430904 (+ 542us) server_negotiation.cc:407] Connection header received
0928 02:52:31.430978 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.430984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.431189 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.431333 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:31.431783 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.433794 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.436250 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.437050 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.437142 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.437280 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.438307 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.438416 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":151,"threads_started":1}
I20260928 02:52:31.440354 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.430338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:44553 (local address 127.0.0.1:41302)
0928 02:52:31.431931 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.431944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.432027 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.433025 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.433029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.433046 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:31.433138 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.433144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.435133 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.435136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.436510 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.436521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.439959 (+ 3438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.439963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.440051 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.440073 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.440127 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.440221 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"thread_start_us":87,"threads_started":1}
I20260928 02:52:31.440919 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.430774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.19.139.130:44553)
0928 02:52:31.432341 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.432351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.432372 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:52:31.432439 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.432444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.432640 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.432782 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:31.433454 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.435009 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.437453 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.438924 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.439030 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.440629 (+ 1599us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.440684 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.440790 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":106,"threads_started":1}
I20260928 02:52:31.441990 31660 tablet_service.cc:1467] Processing CreateTablet for tablet bceb96c9f4634c3c80998b6301b35795 (DEFAULT_TABLE table=test-table [id=e924bbc252b14022b868dab1f8297846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:31.443181 31660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceb96c9f4634c3c80998b6301b35795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:31.444059 31515 tablet_service.cc:1467] Processing CreateTablet for tablet bceb96c9f4634c3c80998b6301b35795 (DEFAULT_TABLE table=test-table [id=e924bbc252b14022b868dab1f8297846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:31.445132 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceb96c9f4634c3c80998b6301b35795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:31.453214 31745 tablet_bootstrap.cc:492] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Bootstrap starting.
I20260928 02:52:31.456377 31745 tablet_bootstrap.cc:654] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:31.458384 31745 log.cc:824] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:31.459149 31746 tablet_bootstrap.cc:492] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Bootstrap starting.
I20260928 02:52:31.459707 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.431532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:37817 (local address 127.0.0.1:37472)
0928 02:52:31.433275 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.433287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.433356 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.437195 (+ 3839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.437198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.437216 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:31.437318 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.437324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.456673 (+ 19349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.456677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.457923 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.457938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.459239 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.459243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.459356 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.459388 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.459436 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.459535 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":100,"threads_started":1}
I20260928 02:52:31.460709 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.431677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.19.139.129:37817)
0928 02:52:31.434464 (+ 2787us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.434471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.434494 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:31.436653 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.436661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.436885 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.437041 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:31.438452 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.456507 (+ 18055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.458096 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.459085 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.460258 (+ 1173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.460398 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.460454 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.460558 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2681,"thread_start_us":80,"threads_started":1}
I20260928 02:52:31.462553 31746 tablet_bootstrap.cc:654] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:31.463688 31745 tablet_bootstrap.cc:492] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: No bootstrap required, opened a new log
I20260928 02:52:31.463846 31746 log.cc:824] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:31.463989 31745 ts_tablet_manager.cc:1397] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260928 02:52:31.464023 31390 tablet_service.cc:1467] Processing CreateTablet for tablet bceb96c9f4634c3c80998b6301b35795 (DEFAULT_TABLE table=test-table [id=e924bbc252b14022b868dab1f8297846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:31.465198 31390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceb96c9f4634c3c80998b6301b35795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:31.468400 31745 raft_consensus.cc:348] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.468724 31745 raft_consensus.cc:374] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:31.468778 31745 raft_consensus.cc:729] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01777e4680144bdaaa63b845b8778bd, State: Initialized, Role: FOLLOWER
I20260928 02:52:31.469296 31745 consensus_queue.cc:260] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.470976 31745 ts_tablet_manager.cc:1428] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:52:31.471334 31727 heartbeater.cc:503] Master 127.19.139.190:40979 was elected leader, sending a full tablet report...
I20260928 02:52:31.475764 31746 tablet_bootstrap.cc:492] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: No bootstrap required, opened a new log
I20260928 02:52:31.476011 31746 ts_tablet_manager.cc:1397] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20260928 02:52:31.480293 31746 raft_consensus.cc:348] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.480701 31746 raft_consensus.cc:374] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:31.480815 31746 raft_consensus.cc:729] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295c0a2ce79f493cab0f7793950d0c1c, State: Initialized, Role: FOLLOWER
I20260928 02:52:31.481410 31746 consensus_queue.cc:260] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.482196 31594 heartbeater.cc:503] Master 127.19.139.190:40979 was elected leader, sending a full tablet report...
I20260928 02:52:31.482570 31746 ts_tablet_manager.cc:1428] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:52:31.483523 31750 tablet_bootstrap.cc:492] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: Bootstrap starting.
I20260928 02:52:31.487602 31750 tablet_bootstrap.cc:654] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:31.488538 31750 log.cc:824] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:31.490796 31750 tablet_bootstrap.cc:492] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: No bootstrap required, opened a new log
I20260928 02:52:31.491006 31750 ts_tablet_manager.cc:1397] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 02:52:31.495932 31750 raft_consensus.cc:348] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.496253 31750 raft_consensus.cc:374] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:31.496371 31750 raft_consensus.cc:729] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93159a5963a44240830d0b98251ae012, State: Initialized, Role: FOLLOWER
I20260928 02:52:31.496850 31750 consensus_queue.cc:260] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.497920 31750 ts_tablet_manager.cc:1428] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 02:52:31.498292 31457 heartbeater.cc:503] Master 127.19.139.190:40979 was elected leader, sending a full tablet report...
W20260928 02:52:31.505565 31595 tablet.cc:2369] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:52:31 dist-test-slave-5vc0 krb5kdc[30767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563950, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
W20260928 02:52:31.621975 31728 tablet.cc:2369] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:52:31.642359 31458 tablet.cc:2369] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:31.780047 31753 raft_consensus.cc:482] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:31.780310 31753 raft_consensus.cc:504] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.781471 31753 leader_election.cc:288] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f01777e4680144bdaaa63b845b8778bd (127.19.139.131:33845), 295c0a2ce79f493cab0f7793950d0c1c (127.19.139.130:44553)
I20260928 02:52:31.785501 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.781819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:39473 (local address 127.19.139.130:44553)
0928 02:52:31.781941 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.781946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.782200 (+ 254us) server_negotiation.cc:407] Connection header received
0928 02:52:31.782348 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.782352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.782398 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.782483 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:31.782759 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.783347 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.784566 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.784977 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.785074 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.785127 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.785297 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.785378 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:52:31.785514 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.781731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:44553 (local address 127.19.139.129:39473)
0928 02:52:31.782097 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.782110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.782227 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.782509 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.782512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.782534 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:31.782633 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.782639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.783444 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.783448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.784440 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.784449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.785101 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.785104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.785188 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.785214 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.785280 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.785339 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":59,"thread_start_us":122,"threads_started":1}
I20260928 02:52:31.785573 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.781819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:50763 (local address 127.19.139.131:33845)
0928 02:52:31.781941 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.781946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.782615 (+ 669us) server_negotiation.cc:407] Connection header received
0928 02:52:31.782698 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.782702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.782752 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.782806 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:31.783037 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.783576 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.784663 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.785078 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.785213 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.785260 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.785401 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.785468 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:52:31.786089 31764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.781731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:33845 (local address 127.19.139.129:50763)
0928 02:52:31.782521 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.782534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:31.782622 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:31.782828 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:31.782830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:31.782847 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:31.782945 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.782951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.783699 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.783702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.784542 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:31.784552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.785185 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.785188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:31.785290 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.785312 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:52:31.785380 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:52:31.785440 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":29,"spinlock_wait_cycles":42240,"thread_start_us":595,"threads_started":1}
I20260928 02:52:31.786152 31543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb96c9f4634c3c80998b6301b35795" candidate_uuid: "93159a5963a44240830d0b98251ae012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "295c0a2ce79f493cab0f7793950d0c1c" is_pre_election: true
I20260928 02:52:31.786460 31543 raft_consensus.cc:2454] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93159a5963a44240830d0b98251ae012 in term 0.
I20260928 02:52:31.786680 31663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb96c9f4634c3c80998b6301b35795" candidate_uuid: "93159a5963a44240830d0b98251ae012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01777e4680144bdaaa63b845b8778bd" is_pre_election: true
I20260928 02:52:31.786947 31663 raft_consensus.cc:2454] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93159a5963a44240830d0b98251ae012 in term 0.
I20260928 02:52:31.786936 31346 leader_election.cc:302] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295c0a2ce79f493cab0f7793950d0c1c, 93159a5963a44240830d0b98251ae012; no voters:
I20260928 02:52:31.787171 31753 raft_consensus.cc:2781] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:31.787259 31753 raft_consensus.cc:482] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:31.787308 31753 raft_consensus.cc:3037] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:31.788344 31753 raft_consensus.cc:504] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.788658 31753 leader_election.cc:288] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [CANDIDATE]: Term 1 election: Requested vote from peers f01777e4680144bdaaa63b845b8778bd (127.19.139.131:33845), 295c0a2ce79f493cab0f7793950d0c1c (127.19.139.130:44553)
I20260928 02:52:31.788972 31663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb96c9f4634c3c80998b6301b35795" candidate_uuid: "93159a5963a44240830d0b98251ae012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01777e4680144bdaaa63b845b8778bd"
I20260928 02:52:31.788977 31543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceb96c9f4634c3c80998b6301b35795" candidate_uuid: "93159a5963a44240830d0b98251ae012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "295c0a2ce79f493cab0f7793950d0c1c"
I20260928 02:52:31.789083 31663 raft_consensus.cc:3037] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:31.789099 31543 raft_consensus.cc:3037] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:31.790051 31543 raft_consensus.cc:2454] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93159a5963a44240830d0b98251ae012 in term 1.
I20260928 02:52:31.790048 31663 raft_consensus.cc:2454] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93159a5963a44240830d0b98251ae012 in term 1.
I20260928 02:52:31.790323 31346 leader_election.cc:302] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295c0a2ce79f493cab0f7793950d0c1c, 93159a5963a44240830d0b98251ae012; no voters:
I20260928 02:52:31.790484 31753 raft_consensus.cc:2781] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:31.790688 31753 raft_consensus.cc:686] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 1 LEADER]: Becoming Leader. State: Replica: 93159a5963a44240830d0b98251ae012, State: Running, Role: LEADER
I20260928 02:52:31.791015 31753 consensus_queue.cc:237] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } }
I20260928 02:52:31.794287 31243 catalog_manager.cc:5495] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93159a5963a44240830d0b98251ae012 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "93159a5963a44240830d0b98251ae012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93159a5963a44240830d0b98251ae012" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 37817 } health_report { overall_health: HEALTHY } } }
I20260928 02:52:31.804280 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.801040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.19.139.129:37817)
0928 02:52:31.801155 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.801158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.801171 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:52:31.801309 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.801311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.801364 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.801446 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:31.801631 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.802105 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.802941 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.803138 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.803216 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.803446 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:31.803859 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:31.803944 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.804045 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.804093 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:52:31.810001 31543 raft_consensus.cc:1261] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 1 FOLLOWER]: Refusing update from remote peer 93159a5963a44240830d0b98251ae012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:31.810307 31663 raft_consensus.cc:1261] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 1 FOLLOWER]: Refusing update from remote peer 93159a5963a44240830d0b98251ae012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:31.810690 31765 consensus_queue.cc:1035] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "295c0a2ce79f493cab0f7793950d0c1c" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:31.810882 31753 consensus_queue.cc:1035] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f01777e4680144bdaaa63b845b8778bd" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:31.819128 31770 mvcc.cc:205] Tried to move back new op lower bound from 7334149946606694400 to 7334149946538770432. Current Snapshot: MvccSnapshot[applied={T|T < 7334149946606694400}]
I20260928 02:52:31.827145 31239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:31.827323 31239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:31.829660 31239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3d12c54d0647028b68a0f28b60d84a: Sending DeleteTablet for 3 replicas of tablet bceb96c9f4634c3c80998b6301b35795
I20260928 02:52:31.830489 31660 tablet_service.cc:1514] Processing DeleteTablet for tablet bceb96c9f4634c3c80998b6301b35795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:31 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54722
I20260928 02:52:31.830493 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet bceb96c9f4634c3c80998b6301b35795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:31 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:41302
I20260928 02:52:31.830516 31390 tablet_service.cc:1514] Processing DeleteTablet for tablet bceb96c9f4634c3c80998b6301b35795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:31 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:37472
I20260928 02:52:31.831099 31778 tablet_replica.cc:330] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: stopping tablet replica
I20260928 02:52:31.831143 31779 tablet_replica.cc:330] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012: stopping tablet replica
I20260928 02:52:31.831395 31780 tablet_replica.cc:330] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: stopping tablet replica
I20260928 02:52:31.831502 31778 raft_consensus.cc:2229] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:31.831740 31779 raft_consensus.cc:2229] T bceb96c9f4634c3c80998b6301b35795 P 93159a5963a44240830d0b98251ae012 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:31.831926 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 31324
I20260928 02:52:31.831957 31778 raft_consensus.cc:2258] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:31.832744 31780 raft_consensus.cc:2229] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:31.833033 31780 raft_consensus.cc:2258] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:31.835449 31780 ts_tablet_manager.cc:1905] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:31.835848 31778 ts_tablet_manager.cc:1905] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:31.837858 31780 ts_tablet_manager.cc:1918] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:31.837953 31780 log.cc:1196] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/bceb96c9f4634c3c80998b6301b35795
I20260928 02:52:31.838299 31780 ts_tablet_manager.cc:1939] T bceb96c9f4634c3c80998b6301b35795 P 295c0a2ce79f493cab0f7793950d0c1c: Deleting consensus metadata
W20260928 02:52:31.838878 31230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:37817 (error 108)
I20260928 02:52:31.838932 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 31463
I20260928 02:52:31.839954 31778 ts_tablet_manager.cc:1918] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:31.840055 31778 log.cc:1196] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/bceb96c9f4634c3c80998b6301b35795
I20260928 02:52:31.840354 31778 ts_tablet_manager.cc:1939] T bceb96c9f4634c3c80998b6301b35795 P f01777e4680144bdaaa63b845b8778bd: Deleting consensus metadata
I20260928 02:52:31.840883 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.840632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:37817 (local address 127.0.0.1:37492)
0928 02:52:31.840749 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.840808 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:37817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:52:31.841028 31230 catalog_manager.cc:4584] TS 93159a5963a44240830d0b98251ae012 (127.19.139.129:37817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bceb96c9f4634c3c80998b6301b35795: Network error: Client connection negotiation failed: client connection to 127.19.139.129:37817: connect: Connection refused (error 111)
I20260928 02:52:31.842334 31227 catalog_manager.cc:4841] TS f01777e4680144bdaaa63b845b8778bd (127.19.139.131:33845): tablet bceb96c9f4634c3c80998b6301b35795 (table test-table [id=e924bbc252b14022b868dab1f8297846]) successfully deleted
I20260928 02:52:31.844364 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 31598
I20260928 02:52:31.848529 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.844518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:44553 (local address 127.0.0.1:41316)
0928 02:52:31.848372 (+ 3854us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:31.848434 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3746}
I20260928 02:52:31.848682 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 31211
I20260928 02:52:31.854363 20014 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:52:32.189503 20014 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:52:32.189635 20014 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:52:32.524567 20014 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:52:32.524 UTC [30797] LOG: received smart shutdown request
2026-09-28 02:52:32.528 UTC [30797] LOG: background worker "logical replication launcher" (PID 30806) exited with exit code 1
2026-09-28 02:52:32.528 UTC [30801] LOG: shutting down
2026-09-28 02:52:32.687 UTC [30797] LOG: database system is shut down
2026-09-28T02:52:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46095 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:32 dist-test-slave-5vc0 krb5kdc[31790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:32 dist-test-slave-5vc0 krb5kdc[31790](info): set up 2 sockets
Sep 28 02:52:32 dist-test-slave-5vc0 krb5kdc[31790](info): commencing operation
krb5kdc: starting...
W20260928 02:52:33.756033 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:52:33 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/rangeradmin_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/rangeradmin_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/rangerlookup_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/rangerlookup_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
I20260928 02:52:33.923426 20014 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAxMa1R/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/postgres -l logfile start
2026-09-28 02:52:36.384 UTC [31820] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:52:36.384 UTC [31820] LOG: listening on IPv4 address "127.19.139.148", port 52155
2026-09-28 02:52:36.388 UTC [31820] LOG: listening on Unix socket "/tmp/.s.PGSQL.52155"
2026-09-28 02:52:36.416 UTC [31823] LOG: database system was shut down at 2026-09-28 02:52:34 UTC
2026-09-28 02:52:36.419 UTC [31820] LOG: database system is ready to accept connections
I20260928 02:52:37.291913 20014 mini_postgres.cc:96] Postgres bound to 52155
2026-09-28 02:52:37.297 UTC [31831] FATAL: database "slave" does not exist
127.19.139.148:52155 - accepting connections
I20260928 02:52:37.298424 20014 mini_ranger.cc:161] Starting Ranger...
I20260928 02:52:37.313104 20014 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:52:37.765465 20014 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:52:37.765534 20014 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-admin
2026-09-28 02:52:38,045 [I] DB FLAVOR :POSTGRES
2026-09-28 02:52:38,046 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:52:38,046 [I] Checking connection..
2026-09-28 02:52:38,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:52:38,309 [I] Checking connection passed.
2026-09-28 02:52:38,309 [I] --------- Verifying version history table ---------
2026-09-28 02:52:38,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:52:38,558 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:52:38,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:52:38,801 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:52:38,801 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:52:38,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:52:39.060 UTC [31895] WARNING: there is no transaction in progress
2026-09-28 02:52:39,066 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:52:39,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:52:39,327 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:52:39,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:52:39,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:52:39,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:52:39,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5vc0.c.gcp-upstream.internal','N') ;"
2026-09-28 02:52:40,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:52:40,333 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:52:40,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:52:40,572 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:52:40,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:52:40,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:52:41,054 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:52:41,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:52:42.024 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.040 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.057 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.072 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.089 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.111 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.166 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.173 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.188 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.198 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.208 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.222 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.232 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.239 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.246 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42.253 UTC [32003] WARNING: there is no transaction in progress
2026-09-28 02:52:42,932 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:52:42,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:52:43,165 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:52:43,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:52:43,404 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:52:43,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:52:43,650 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:52:43,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:52:43,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:52:44,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5vc0.c.gcp-upstream.internal';"
2026-09-28 02:52:44,356 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:52:44,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:52:44,613 [I] DB_PATCHES have already been applied
I20260928 02:52:44.619105 20014 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAxMa1R/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:52:44.619186 20014 mini_ranger.cc:193] Using host: 127.19.139.148
I20260928 02:52:44.622457 20014 mini_ranger.cc:227] Ranger admin URL: http://127.19.139.148:51939
Sep 28, 2026 2:52:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:52:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:52:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:52:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:52:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:52:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:52:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:52:45.596724 20014 mini_ranger.cc:160] Time spent starting Ranger: real 8.298s user 0.000s sys 0.009s
Sep 28, 2026 2:53:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:53:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:53:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:53:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:53:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:53:08.077894 20014 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.139.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.139.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/rangerkms_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.139.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/rangerkms_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.139.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.139.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.139.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/HTTP_127.19.139.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:53:08 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:53:08.167106 20014 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:53:08.177400 20014 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:53:09.045347 20014 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:53:09.045430 20014 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms
I20260928 02:53:09.045447 20014 mini_ranger_kms.cc:227] Using postgres at 127.19.139.148:52155
2026-09-28 02:53:09,217 [I] DB FLAVOR :POSTGRES
2026-09-28 02:53:09,218 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:53:09,218 [I] Checking connection
2026-09-28 02:53:09,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:53:09,489 [I] connection success
2026-09-28 02:53:09,489 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:53:09,489 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:53:09,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:53:09,795 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:53:09,795 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:53:09,795 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:53:09,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.139.148:52155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:53:10,098 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:53:10.588939 20014 mini_ranger_kms.cc:310] Created kms service
I20260928 02:53:10.841886 20014 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:53:10.887298 20014 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:53:10 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:11.287575 20014 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:53:11.287842 20014 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAxMa1R/build/debug/bin/postgresql.jar:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAxMa1R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:53:11.287871 20014 mini_ranger_kms.cc:240] Using host: 127.19.139.148
I20260928 02:53:11.291558 20014 mini_ranger_kms.cc:276] Ranger KMS PID: 32183
I20260928 02:53:11.291649 20014 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.139.148:60275
Sep 28, 2026 2:53:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:53:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:53:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:53:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:53:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:53:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:53:13.307518 20014 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.140s user 0.000s sys 0.157s
I20260928 02:53:13.307654 20014 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:53:13.307726 20014 mini_ranger_kms.cc:401] 127.19.139.148:60275/kms/v1/keys
Sep 28 02:53:13 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563993, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.139.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:53:13 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563993, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.139.148@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
Sep 28 02:53:14 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563993, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.139.148@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
Sep 28 02:53:16 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:53:16.612519 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:44483
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:44483
--ranger_config_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:16.730897 32234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:16.731217 32234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:16.731282 32234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:16.735503 32234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:16.735597 32234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:16.735637 32234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:16.735664 32234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:16.735697 32234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:16.741535 32234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.139.148:60275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:44483
--ranger_config_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:16.742728 32234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:16.744139 32234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:16.753984 32242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:16.754012 32239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:16.754153 32234 server_base.cc:1034] running on GCE node
W20260928 02:53:16.753990 32240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:16.755054 32234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:16.756479 32234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:16.757648 32234 hybrid_clock.cc:648] HybridClock initialized: now 1790563996757624 us; error 40 us; skew 500 ppm
Sep 28 02:53:16 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563996, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:16.760876 32234 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:16.762034 32234 webserver.cc:460] Webserver started at http://127.19.139.190:38715/ using document root <none> and password file <none>
I20260928 02:53:16.762727 32234 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:16.762804 32234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:16.763054 32234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:53:16 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563996, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:16.828912 32234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "8bf384197772445785307727aa875741"
format_stamp: "Formatted at 2026-09-28 02:53:16 on dist-test-slave-5vc0"
server_key: "37dee0007b75876b757e37d77d5a2b03"
server_key_iv: "c593ee03c5fa6232d0a99903eeb0470b"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:16.829536 32234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "8bf384197772445785307727aa875741"
format_stamp: "Formatted at 2026-09-28 02:53:16 on dist-test-slave-5vc0"
server_key: "37dee0007b75876b757e37d77d5a2b03"
server_key_iv: "c593ee03c5fa6232d0a99903eeb0470b"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:16.833570 32234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:53:16.852344 32249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:16.853750 32234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:53:16.853878 32234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "8bf384197772445785307727aa875741"
format_stamp: "Formatted at 2026-09-28 02:53:16 on dist-test-slave-5vc0"
server_key: "37dee0007b75876b757e37d77d5a2b03"
server_key_iv: "c593ee03c5fa6232d0a99903eeb0470b"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:16.854012 32234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:16.864579 32234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:16.867328 32234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:16.867512 32234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:16.875144 32234 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:44483
I20260928 02:53:16.875169 32301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:44483 every 8 connection(s)
I20260928 02:53:16.876191 32234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:16.879652 32302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:16.881016 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 32234
I20260928 02:53:16.881124 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:53:16.881409 20014 external_mini_cluster.cc:1428] Setting key 1df4ca2a515fad415f541dfd57700129
I20260928 02:53:16.887004 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Bootstrap starting.
I20260928 02:53:16.889776 32302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Neither blocks nor log segments found. Creating new log.
Sep 28 02:53:16 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:16.890666 32302 log.cc:824] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:16.894155 32302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: No bootstrap required, opened a new log
I20260928 02:53:16.895886 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:16.882875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.19.139.190:44483)
0928 02:53:16.883271 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:16.883286 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:16.883324 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:53:16.884041 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:16.884061 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:16.884472 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:16.884798 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:16.885579 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.886558 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.887457 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.887935 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.888081 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.890459 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:16.890465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:16.890479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:16.890507 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:16.892594 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:16.893152 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:16.893157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:16.893163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:16.893299 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:16.893606 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:16.893609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:16.893611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:16.894126 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:16.894434 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:16.894593 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:53:16.894903 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
I20260928 02:53:16.898180 32302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:16.898473 32302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:16.898525 32302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf384197772445785307727aa875741, State: Initialized, Role: FOLLOWER
I20260928 02:53:16.898996 32302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:16.899127 32302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:16.899183 32302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:16.899277 32302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:16.900300 32302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:16.900631 32302 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bf384197772445785307727aa875741; no voters:
I20260928 02:53:16.900946 32302 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:16.901029 32307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:16.901185 32307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 LEADER]: Becoming Leader. State: Replica: 8bf384197772445785307727aa875741, State: Running, Role: LEADER
I20260928 02:53:16.901491 32307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:16.902252 32302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:16.902657 32307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bf384197772445785307727aa875741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } } }
I20260928 02:53:16.902781 32307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:16.902976 32307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf384197772445785307727aa875741. Latest consensus state: current_term: 1 leader_uuid: "8bf384197772445785307727aa875741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } } }
I20260928 02:53:16.903064 32307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:16.904351 32302 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:53:16.905555 32311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:16.911252 32311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:16.918573 32311 catalog_manager.cc:1347] Generated new cluster ID: 88052431f5144ecd81f6f3366acbdd9d
I20260928 02:53:16.918658 32311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:16.939368 32311 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:16.940815 32311 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:16.955235 32311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Generated new TSK 0
I20260928 02:53:16.956236 32311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:53:17.625283 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:44483
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:17.742515 32347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:17.742789 32347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:17.742847 32347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:17.747543 32347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:17.747619 32347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:17.747705 32347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:53:17.753891 32347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.139.148:60275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:17.755034 32347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:17.756093 32347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:17.766283 32353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:17.766422 32355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:17.766294 32352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:17.767601 32347 server_base.cc:1034] running on GCE node
I20260928 02:53:17.768118 32347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:17.769064 32347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:17.770296 32347 hybrid_clock.cc:648] HybridClock initialized: now 1790563997770261 us; error 47 us; skew 500 ppm
Sep 28 02:53:17 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:17.773944 32347 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:53:17.775193 32347 webserver.cc:460] Webserver started at http://127.19.139.129:34087/ using document root <none> and password file <none>
I20260928 02:53:17.775975 32347 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:17.776078 32347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:17.776469 32347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:53:17 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:17.792500 32347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "4a681751e1e94032b78de71ed2bf6175"
format_stamp: "Formatted at 2026-09-28 02:53:17 on dist-test-slave-5vc0"
server_key: "39b8e49ca8debcc3d1b04cd672045226"
server_key_iv: "1efc45ae26a7195f1de589f174c13fe3"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:17.793273 32347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "4a681751e1e94032b78de71ed2bf6175"
format_stamp: "Formatted at 2026-09-28 02:53:17 on dist-test-slave-5vc0"
server_key: "39b8e49ca8debcc3d1b04cd672045226"
server_key_iv: "1efc45ae26a7195f1de589f174c13fe3"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:17.797127 32347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:53:17.821593 32364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:17.823515 32347 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.004s
I20260928 02:53:17.823668 32347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "4a681751e1e94032b78de71ed2bf6175"
format_stamp: "Formatted at 2026-09-28 02:53:17 on dist-test-slave-5vc0"
server_key: "39b8e49ca8debcc3d1b04cd672045226"
server_key_iv: "1efc45ae26a7195f1de589f174c13fe3"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:17.823817 32347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:17.851859 32347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:17.855983 32347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:17.856310 32347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:17.857213 32347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:17.858477 32347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:17.858608 32347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:53:17.858748 32347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:17.858834 32347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:17.873181 32347 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:35809
I20260928 02:53:17.874539 32347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:17.882957 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 32347
I20260928 02:53:17.883088 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:17.883404 20014 external_mini_cluster.cc:1428] Setting key 1392ceb682f496e9fb9a66fc582e780c
I20260928 02:53:17.888357 32477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:35809 every 8 connection(s)
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Sep 28 02:53:17 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:17.926307 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:17.891045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:44483 (local address 127.19.139.129:44119)
0928 02:53:17.894678 (+ 3633us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:17.894743 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:17.895862 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:17.900725 (+ 4863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:17.900737 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:17.901102 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:17.901702 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.901723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.902885 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.902888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.904309 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.904321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.905041 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.905044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.905249 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.906104 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:17.906126 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:17.907698 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:17.922452 (+ 14754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:17.922462 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:17.922485 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:17.922962 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:17.923369 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:17.923373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:17.923375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:17.923620 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:17.924181 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:17.924190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:17.924556 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 02:53:17.924765 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 02:53:17.925172 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3368,"thread_start_us":193,"threads_started":1}
I20260928 02:53:17.927804 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:17.892293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:44119 (local address 127.19.139.190:44483)
0928 02:53:17.900179 (+ 7886us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:17.900188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:17.900208 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:53:17.900257 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:17.900261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:17.900332 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:17.901098 (+ 766us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:17.901888 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.902741 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.904450 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.904756 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.904860 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.907918 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:17.907922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:17.907924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:17.907953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:17.922241 (+ 14288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:17.923124 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:17.923129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:17.923131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:17.923248 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:17.923754 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:17.923757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:17.923759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:17.924029 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:17.927359 (+ 3330us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:17.927458 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:53:17.927613 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7756,"thread_start_us":132,"threads_started":1}
I20260928 02:53:17.929205 32478 heartbeater.cc:347] Connected to a master server at 127.19.139.190:44483
I20260928 02:53:17.929469 32478 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:17.930119 32478 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:53:17.932102 32264 ts_manager.cc:194] Registered new tserver with Master: 4a681751e1e94032b78de71ed2bf6175 (127.19.139.129:35809)
I20260928 02:53:17.932451 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:44483
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:17.933522 32264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:44119
Sep 28 02:53:17 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:53:18.103611 32486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:18.104503 32486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:18.104645 32486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:18.111115 32486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:18.111347 32486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:18.111509 32486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:53:18.119544 32486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.139.148:60275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:18.121193 32486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:18.122576 32486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:18.135358 32493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:18.135407 32492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:18.141084 32486 server_base.cc:1034] running on GCE node
W20260928 02:53:18.141505 32495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:18.142117 32486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:18.143038 32486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:18.144342 32486 hybrid_clock.cc:648] HybridClock initialized: now 1790563998144275 us; error 90 us; skew 500 ppm
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:18.152806 32486 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:53:18.154045 32486 webserver.cc:460] Webserver started at http://127.19.139.130:41577/ using document root <none> and password file <none>
I20260928 02:53:18.154810 32486 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:18.154945 32486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:18.155225 32486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:18.187754 32486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "a3cfdf89087749e2a2173471121082c0"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "1c968e2fb28fb59c363c969afb87b647"
server_key_iv: "b6052781f594b3b0fd6a16042e1ff5fd"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.188642 32486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "a3cfdf89087749e2a2173471121082c0"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "1c968e2fb28fb59c363c969afb87b647"
server_key_iv: "b6052781f594b3b0fd6a16042e1ff5fd"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.193838 32486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:53:18.225353 32502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.227180 32486 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260928 02:53:18.227324 32486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "a3cfdf89087749e2a2173471121082c0"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "1c968e2fb28fb59c363c969afb87b647"
server_key_iv: "b6052781f594b3b0fd6a16042e1ff5fd"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.227456 32486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:18.245411 32486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:18.248952 32486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:18.249240 32486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:18.250059 32486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:18.251081 32486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:18.251195 32486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.251287 32486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:18.251343 32486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.261804 32486 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:35673
I20260928 02:53:18.264298 32486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:18.270615 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 32486
I20260928 02:53:18.270731 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:18.271018 20014 external_mini_cluster.cc:1428] Setting key 36bca40598a59fb61c16bcb0d1ad9c6d
I20260928 02:53:18.267117 32615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:35673 every 8 connection(s)
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:18.294272 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.271439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:39163 (local address 127.19.139.190:44483)
0928 02:53:18.271805 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:18.271810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:18.275524 (+ 3714us) server_negotiation.cc:407] Connection header received
0928 02:53:18.276783 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:18.276787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:18.276841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:18.276888 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:18.278517 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.279323 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.280974 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.281323 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.281394 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.286305 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:18.286309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:18.286312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:18.286341 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:18.288032 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:18.292289 (+ 4257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:18.292294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:18.292296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:18.292406 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:18.292873 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:18.292877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:18.292879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:18.293112 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:18.293206 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:18.293923 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 02:53:18.294066 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260928 02:53:18.296219 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.269340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:44483 (local address 127.19.139.130:39163)
0928 02:53:18.275373 (+ 6033us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:18.275429 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:18.276530 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:18.277316 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:18.277326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:18.277681 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:18.278351 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.278369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.279476 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.279479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.280845 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.280856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.281613 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.281617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.281819 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.284070 (+ 2251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:18.284092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:18.286082 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:18.288243 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:18.288250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:18.288270 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:18.288614 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:18.292526 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:18.292529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:18.292532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:18.292759 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:18.293278 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:18.293285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:18.293641 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 02:53:18.294738 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0928 02:53:18.295112 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":148,"threads_started":1}
I20260928 02:53:18.297811 32616 heartbeater.cc:347] Connected to a master server at 127.19.139.190:44483
I20260928 02:53:18.298053 32616 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:18.298622 32616 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:18.299844 32264 ts_manager.cc:194] Registered new tserver with Master: a3cfdf89087749e2a2173471121082c0 (127.19.139.130:35673)
I20260928 02:53:18.300632 32264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:39163
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:53:18.327450 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:44483
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563997, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
W20260928 02:53:18.517769 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:18.518285 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:18.518409 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:18.524562 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:18.524787 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:18.525027 32621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:53:18.532958 32621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.139.148:60275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:18.534605 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:18.536067 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:18.550470 32627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:18.550537 32626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:18.551716 32621 server_base.cc:1034] running on GCE node
W20260928 02:53:18.559646 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:18.560462 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:18.561498 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:18.562974 32621 hybrid_clock.cc:648] HybridClock initialized: now 1790563998562905 us; error 100 us; skew 500 ppm
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:18.566841 32621 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:53:18.568092 32621 webserver.cc:460] Webserver started at http://127.19.139.131:38633/ using document root <none> and password file <none>
I20260928 02:53:18.568958 32621 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:18.569118 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:18.569666 32621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:18.590106 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "e4d64ca43a2c48ebaabf760e75222b47"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "82288963bb659573583df06e79be84c6"
server_key_iv: "3f8e38f013e054ae07081718a12bfc19"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.591009 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "e4d64ca43a2c48ebaabf760e75222b47"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "82288963bb659573583df06e79be84c6"
server_key_iv: "3f8e38f013e054ae07081718a12bfc19"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.601794 32621 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260928 02:53:18.629621 32636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.631593 32621 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.000s
I20260928 02:53:18.631772 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "e4d64ca43a2c48ebaabf760e75222b47"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "82288963bb659573583df06e79be84c6"
server_key_iv: "3f8e38f013e054ae07081718a12bfc19"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.631922 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:18.646550 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:18.649513 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:18.649699 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:18.650329 32621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:18.651221 32621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:18.651269 32621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.651322 32621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:18.651377 32621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.661684 32621 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:36591
I20260928 02:53:18.661723 32749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:36591 every 8 connection(s)
I20260928 02:53:18.662760 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:18.666633 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 32621
I20260928 02:53:18.666785 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:18.667111 20014 external_mini_cluster.cc:1428] Setting key a802a349914fbf597217da445394aeec
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:18.679219 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.665645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:46531 (local address 127.19.139.190:44483)
0928 02:53:18.665781 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:18.665784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:18.667913 (+ 2129us) server_negotiation.cc:407] Connection header received
0928 02:53:18.669383 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:18.669386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:18.669443 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:18.669493 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:18.670889 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.671428 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.672987 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.673223 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.673315 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.675749 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:18.675753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:18.675755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:18.675782 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:18.677187 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:18.677703 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:18.677706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:18.677707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:18.677784 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:18.678116 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:18.678120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:18.678121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:18.678311 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:18.678395 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:18.678926 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 02:53:18.679051 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:53:18.680596 32752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.666392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:44483 (local address 127.19.139.131:46531)
0928 02:53:18.667756 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:18.667816 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:18.669137 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:18.669918 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:18.669925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:18.670214 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:18.670730 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.670744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.671559 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.671563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.672851 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.672885 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.673378 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.673382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.673565 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.674153 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:18.674171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:18.675503 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:18.677307 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:18.677313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:18.677331 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:18.677599 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:18.677871 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:18.677874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:18.677876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:18.678026 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:18.678412 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:18.678419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:18.678692 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 02:53:18.678868 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 02:53:18.679295 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"thread_start_us":155,"threads_started":1}
I20260928 02:53:18.682018 32750 heartbeater.cc:347] Connected to a master server at 127.19.139.190:44483
I20260928 02:53:18.682276 32750 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:18.682798 32750 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:18.683800 32264 ts_manager.cc:194] Registered new tserver with Master: e4d64ca43a2c48ebaabf760e75222b47 (127.19.139.131:36591)
I20260928 02:53:18.684406 32264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:46531
I20260928 02:53:18.693678 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:53:18.694432 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 32347
I20260928 02:53:18.699736 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 32486
I20260928 02:53:18.706315 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 32621
I20260928 02:53:18.713392 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 32234
I20260928 02:53:18.719323 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:44483
--webserver_interface=127.19.139.190
--webserver_port=38715
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:44483
--ranger_config_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:18.829926 32753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:18.830214 32753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:18.830269 32753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:18.834308 32753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:18.834378 32753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:18.834405 32753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:18.834426 32753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:18.834447 32753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:18.839985 32753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:44483
--ranger_config_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:18.841120 32753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:18.842139 32753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:53:18.850646 32753 server_base.cc:1034] running on GCE node
W20260928 02:53:18.850608 32758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:18.850608 32761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:18.850608 32759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:18.851438 32753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:18.852617 32753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:18.853816 32753 hybrid_clock.cc:648] HybridClock initialized: now 1790563998853790 us; error 44 us; skew 500 ppm
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:18.856664 32753 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:18.857636 32753 webserver.cc:460] Webserver started at http://127.19.139.190:38715/ using document root <none> and password file <none>
I20260928 02:53:18.858170 32753 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:18.858244 32753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:53:18 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:18.920719 32753 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:53:18.923198 301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:18.924669 32753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:18.924804 32753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "8bf384197772445785307727aa875741"
format_stamp: "Formatted at 2026-09-28 02:53:16 on dist-test-slave-5vc0"
server_key: "37dee0007b75876b757e37d77d5a2b03"
server_key_iv: "c593ee03c5fa6232d0a99903eeb0470b"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:18.925415 32753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:18.946535 32753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:18.949275 32753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:18.949441 32753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:18.956605 32753 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:44483
I20260928 02:53:18.956701 353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:44483 every 8 connection(s)
I20260928 02:53:18.957787 32753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:18.962200 354 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:53:18.964046 354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Bootstrap starting.
I20260928 02:53:18.967000 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 32753
I20260928 02:53:18.967456 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:35809
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:44483
--webserver_port=34087
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:18.981025 354 log.cc:824] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:18.988006 354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:53:18.988461 354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Bootstrap complete.
I20260928 02:53:18.992522 354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:18.992962 354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf384197772445785307727aa875741, State: Initialized, Role: FOLLOWER
I20260928 02:53:18.993490 354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:18.993616 354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:18.993686 354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:18.993783 354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:53:18.995782 354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:18.996116 354 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bf384197772445785307727aa875741; no voters:
I20260928 02:53:18.996523 354 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:53:18.996616 358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:53:18.996824 358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [term 2 LEADER]: Becoming Leader. State: Replica: 8bf384197772445785307727aa875741, State: Running, Role: LEADER
I20260928 02:53:18.997196 358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } }
I20260928 02:53:18.997823 354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:18.998669 360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf384197772445785307727aa875741. Latest consensus state: current_term: 2 leader_uuid: "8bf384197772445785307727aa875741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } } }
I20260928 02:53:18.998791 360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:18.999027 359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8bf384197772445785307727aa875741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf384197772445785307727aa875741" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44483 } } }
I20260928 02:53:18.999097 359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:19.000111 354 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:53:19.000641 367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:19.006765 367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:19.007665 367 catalog_manager.cc:1259] Loaded cluster ID: 88052431f5144ecd81f6f3366acbdd9d
I20260928 02:53:19.007814 367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:19.010433 367 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:19.011199 367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Loaded TSK: 0
I20260928 02:53:19.012579 367 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:53:19.109201 356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:19.109647 356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:19.109740 356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:19.116034 356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:19.116221 356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:19.116353 356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:53:19.122996 356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:35809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=34087
--tserver_master_addrs=127.19.139.190:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:19.124305 356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:19.125397 356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:19.135659 392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:19.135659 394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:19.136061 391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:19.136207 356 server_base.cc:1034] running on GCE node
I20260928 02:53:19.136660 356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:19.137331 356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:19.138504 356 hybrid_clock.cc:648] HybridClock initialized: now 1790563999138481 us; error 41 us; skew 500 ppm
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:19.141497 356 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:53:19.142807 356 webserver.cc:460] Webserver started at http://127.19.139.129:34087/ using document root <none> and password file <none>
I20260928 02:53:19.143580 356 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:19.143674 356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:19.166819 356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:53:19.170157 402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.171793 356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 02:53:19.171931 356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "4a681751e1e94032b78de71ed2bf6175"
format_stamp: "Formatted at 2026-09-28 02:53:17 on dist-test-slave-5vc0"
server_key: "39b8e49ca8debcc3d1b04cd672045226"
server_key_iv: "1efc45ae26a7195f1de589f174c13fe3"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:19.172683 356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:19.187731 356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:19.190840 356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:19.191008 356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:19.191612 356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:19.192550 356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:19.192598 356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.192637 356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:19.192708 356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.203150 356 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:35809
I20260928 02:53:19.204615 356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:19.205245 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 356
I20260928 02:53:19.205822 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:35673
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:44483
--webserver_port=41577
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:19.207248 515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:35809 every 8 connection(s)
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:19.241406 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.210744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:44483 (local address 127.19.139.129:43533)
0928 02:53:19.220601 (+ 9857us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:19.220689 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:19.221851 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:19.224397 (+ 2546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:19.224407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:19.224782 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:19.225480 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.225504 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.227531 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.227535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.230257 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.230272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.231640 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.231645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.231850 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.232610 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:19.232633 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:19.234474 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:19.237212 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:19.237220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:19.237249 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:19.237619 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:19.238063 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:19.238066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:19.238068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:19.238274 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:19.239143 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:19.239149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:19.239504 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 02:53:19.239705 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 02:53:19.240075 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9522,"thread_start_us":176,"threads_started":1}
I20260928 02:53:19.243813 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.212353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:43533 (local address 127.19.139.190:44483)
0928 02:53:19.222486 (+ 10133us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.222514 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.222568 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:53:19.222919 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.222939 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.223465 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.223863 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.225782 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.227370 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.230416 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.231190 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.231389 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.234790 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:19.234798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:19.234818 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:19.234855 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:19.237059 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.237771 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.237778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.237785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.237947 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.238388 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.238391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.238393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.238953 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:19.239087 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:19.242442 (+ 3355us) server_negotiation.cc:299] Negotiation successful
0928 02:53:19.242791 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9923,"thread_start_us":118,"threads_started":1}
I20260928 02:53:19.247289 516 heartbeater.cc:347] Connected to a master server at 127.19.139.190:44483
I20260928 02:53:19.247555 516 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.248212 516 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
W20260928 02:53:19.250703 516 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:44483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:53:19.250842 516 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.251042 516 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:19.252704 516 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.252915 516 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
W20260928 02:53:19.253744 516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:53:19.359797 518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:19.360073 518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:19.360136 518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:19.364519 518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:19.364586 518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:19.364648 518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:53:19.370240 518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:35673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=41577
--tserver_master_addrs=127.19.139.190:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:19.371578 518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:19.372862 518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:19.383982 525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:19.384363 528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:19.384503 526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:19.385375 518 server_base.cc:1034] running on GCE node
I20260928 02:53:19.385862 518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:19.386539 518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:19.387774 518 hybrid_clock.cc:648] HybridClock initialized: now 1790563999387735 us; error 56 us; skew 500 ppm
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:19.390825 518 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:53:19.391884 518 webserver.cc:460] Webserver started at http://127.19.139.130:41577/ using document root <none> and password file <none>
I20260928 02:53:19.392509 518 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:19.392596 518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:19.414229 518 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:53:19.417742 536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.419426 518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:53:19.419553 518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "a3cfdf89087749e2a2173471121082c0"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "1c968e2fb28fb59c363c969afb87b647"
server_key_iv: "b6052781f594b3b0fd6a16042e1ff5fd"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:19.420359 518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:19.441300 518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:19.444968 518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:19.445256 518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:19.446053 518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:19.447252 518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:19.447371 518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.447477 518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:19.447799 518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.469063 518 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:35673
I20260928 02:53:19.469362 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:35673 every 8 connection(s)
I20260928 02:53:19.470454 518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:19.477152 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 518
I20260928 02:53:19.477676 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.139.148:60275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:36591
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:44483
--webserver_port=38633
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:19.488147 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.472921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:59277 (local address 127.19.139.190:44483)
0928 02:53:19.473068 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.473073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.473810 (+ 737us) server_negotiation.cc:407] Connection header received
0928 02:53:19.475117 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.475122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.475194 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.475254 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.476957 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.477729 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.479639 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.480014 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.480104 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.484207 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:19.484212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:19.484215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:19.484247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:19.485612 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.486193 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.486197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.486199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.486281 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.486645 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.486649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.486650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.486936 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:19.487031 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:19.487817 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 02:53:19.487968 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:53:19.490450 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.473066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:44483 (local address 127.19.139.130:59277)
0928 02:53:19.473618 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:19.473676 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:19.474823 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:19.475727 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:19.475737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:19.476037 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:19.476760 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.476783 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.478132 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.478136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.479474 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.479488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.480361 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.480366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.480585 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.481427 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:19.481450 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:19.483953 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:19.485763 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:19.485770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:19.485789 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:19.486082 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:19.486375 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:19.486378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:19.486380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:19.486545 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:19.487087 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:19.487094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:19.487469 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 02:53:19.488864 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0928 02:53:19.489271 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":130,"threads_started":1}
I20260928 02:53:19.492271 650 heartbeater.cc:347] Connected to a master server at 127.19.139.190:44483
I20260928 02:53:19.492551 650 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.493183 650 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
W20260928 02:53:19.495132 650 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:44483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:53:19.495262 650 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.495446 650 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:19.496466 650 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.496649 650 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
W20260928 02:53:19.497486 650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:53:19.644048 653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:19.644410 653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:19.644474 653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:19.649098 653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:19.649166 653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:19.649248 653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:53:19.655081 653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.139.148:60275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:36591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=38633
--tserver_master_addrs=127.19.139.190:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:19.656572 653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:19.657920 653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:19.668584 660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:19.668538 659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:19.668881 653 server_base.cc:1034] running on GCE node
W20260928 02:53:19.669061 662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:19.669559 653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:19.670478 653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:19.671701 653 hybrid_clock.cc:648] HybridClock initialized: now 1790563999671679 us; error 34 us; skew 500 ppm
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:19.675346 653 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:53:19.676633 653 webserver.cc:460] Webserver started at http://127.19.139.131:38633/ using document root <none> and password file <none>
I20260928 02:53:19.677398 653 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:19.677515 653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:19.697296 653 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:53:19.700814 670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.702677 653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:53:19.702842 653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "e4d64ca43a2c48ebaabf760e75222b47"
format_stamp: "Formatted at 2026-09-28 02:53:18 on dist-test-slave-5vc0"
server_key: "82288963bb659573583df06e79be84c6"
server_key_iv: "3f8e38f013e054ae07081718a12bfc19"
server_key_version: "kuduclusterkey@0"
I20260928 02:53:19.703671 653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:19.724021 653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:19.727952 653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:19.728235 653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:19.729146 653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:19.730360 653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:19.730432 653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.730494 653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:19.730530 653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:19.742974 653 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:36591
I20260928 02:53:19.743010 783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:36591 every 8 connection(s)
I20260928 02:53:19.744663 653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:19.747818 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 653
Sep 28 02:53:19 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:19.761881 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.746982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:57771 (local address 127.19.139.190:44483)
0928 02:53:19.747108 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.747113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.749633 (+ 2520us) server_negotiation.cc:407] Connection header received
0928 02:53:19.750578 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.750582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.750638 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.750715 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.752004 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.752779 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.755000 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.755447 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.755557 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.758300 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:19.758303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:19.758306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:19.758334 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:19.759650 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.760182 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.760186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.760188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.760277 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.760611 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.760614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.760616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.760808 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:19.760892 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:19.761550 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 02:53:19.761690 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:53:19.762856 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.747570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:44483 (local address 127.19.139.131:57771)
0928 02:53:19.749483 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:19.749541 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:19.750359 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:19.750887 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:19.750895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:19.751226 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:19.751858 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.751873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.752911 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.752914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.754783 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.754795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.755821 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.755826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.756044 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.756790 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:19.756807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:19.758119 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:19.759768 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:19.759774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:19.759792 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:19.760054 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:19.760363 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:19.760366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:19.760368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:19.760521 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:19.760909 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:19.760916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:19.761231 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 02:53:19.761459 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 02:53:19.761837 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"thread_start_us":166,"threads_started":1}
I20260928 02:53:19.764432 784 heartbeater.cc:347] Connected to a master server at 127.19.139.190:44483
I20260928 02:53:19.764700 784 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.765277 784 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
W20260928 02:53:19.767144 784 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:44483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:53:19.767283 784 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.767486 784 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:19.768420 784 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:19.768746 784 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
W20260928 02:53:19.769618 784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:53:19.778337 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.749053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.19.139.190:44483)
0928 02:53:19.760842 (+ 11789us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.760845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.760862 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:53:19.760899 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.760901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.760946 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.761044 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.764248 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.765049 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.767774 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.768203 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.768328 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.774448 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:19.774454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:19.774457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:19.774495 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:19.776251 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.776870 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.776874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.776876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.776981 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:19.777333 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:19.777337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:19.777339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:19.777590 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:19.777751 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:19.778043 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:53:19.778168 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11694,"thread_start_us":121,"threads_started":1}
Sep 28 02:53:20 dist-test-slave-5vc0 krb5kdc[31790](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564000, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:20.253973 516 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:20.254258 516 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:20.255450 318 ts_manager.cc:194] Registered new tserver with Master: 4a681751e1e94032b78de71ed2bf6175 (127.19.139.129:35809)
I20260928 02:53:20.257222 318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:43533
I20260928 02:53:20.497723 650 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:20.498054 650 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:20.498941 317 ts_manager.cc:194] Registered new tserver with Master: a3cfdf89087749e2a2173471121082c0 (127.19.139.130:35673)
I20260928 02:53:20.499779 317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:59277
Sep 28 02:53:20 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564000, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:20.769929 784 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:20.770243 784 heartbeater.cc:511] Master 127.19.139.190:44483 requested a full tablet report, sending...
I20260928 02:53:20.771198 317 ts_manager.cc:194] Registered new tserver with Master: e4d64ca43a2c48ebaabf760e75222b47 (127.19.139.131:36591)
I20260928 02:53:20.771991 317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:57771
Sep 28 02:53:20 dist-test-slave-5vc0 krb5kdc[31790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564000, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for HTTP/127.19.139.148@KRBTEST.COM
I20260928 02:53:20.776113 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:53:20.789311 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.779039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.19.139.190:44483)
0928 02:53:20.779694 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.779699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.780018 (+ 319us) server_negotiation.cc:407] Connection header received
0928 02:53:20.780061 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.780064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.780113 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.780218 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:20.781118 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.781921 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.783023 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.783356 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.783452 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.785266 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:20.785270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:20.785272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:20.785298 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:20.786969 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:20.787487 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:20.787490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:20.787492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:20.787588 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:20.787908 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:20.787912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:20.787913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:20.788191 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:20.788322 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:20.788996 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 02:53:20.789133 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":81,"threads_started":1}
I20260928 02:53:20.793682 317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:20.798502 317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:20.849615 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.833461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35673 (local address 127.0.0.1:40250)
0928 02:53:20.834930 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.834940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.834996 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.845098 (+ 10102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.845101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.845120 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:20.845229 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.845234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.846607 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.846611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.847648 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.847659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.848997 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.849000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.849126 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.849209 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:53:20.849316 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:53:20.849453 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":88,"threads_started":1}
I20260928 02:53:20.850487 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.833606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.19.139.130:35673)
0928 02:53:20.844397 (+ 10791us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.844404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.844432 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:53:20.844515 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.844521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.844781 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.844944 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:20.845347 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.846480 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.847783 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.848664 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.848782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.848930 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:20.850247 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0928 02:53:20.850354 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10656,"thread_start_us":72,"threads_started":1}
I20260928 02:53:20.851166 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.832220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:35809 (local address 127.0.0.1:54418)
0928 02:53:20.834441 (+ 2221us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.834504 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.834635 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.835869 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.835875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.835898 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:20.836042 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.836048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.845509 (+ 9461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.845511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.847002 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.847013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.850784 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.850788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.850873 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.850894 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:53:20.850960 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:53:20.851042 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"thread_start_us":134,"threads_started":1}
I20260928 02:53:20.853739 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.832987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.19.139.131:36591)
0928 02:53:20.837756 (+ 4769us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.837764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.837802 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:53:20.837881 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.837886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.838103 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.838263 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:20.843840 (+ 5577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.845098 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.849186 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.852545 (+ 3359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.853291 (+ 746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.853437 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:20.853494 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:53:20.853596 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4647,"thread_start_us":69,"threads_started":1}
I20260928 02:53:20.853809 584 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2c78c9da22433582c15e2f100fedee (DEFAULT_TABLE table=test-table [id=4719faf423114e90957424c382e3374a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:20.855424 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.833093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.19.139.129:35809)
0928 02:53:20.835218 (+ 2125us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.835225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.835250 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:53:20.835325 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.835331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.835551 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.835702 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:20.836210 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.845386 (+ 9176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.847387 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.850451 (+ 3064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.850574 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.850718 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:20.855153 (+ 4435us) server_negotiation.cc:299] Negotiation successful
0928 02:53:20.855280 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":66,"threads_started":1}
I20260928 02:53:20.856047 584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2c78c9da22433582c15e2f100fedee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:20.853047 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.832633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:36591 (local address 127.0.0.1:54020)
0928 02:53:20.834685 (+ 2052us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.834699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.834780 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.841145 (+ 6365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.841148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.841172 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:20.841294 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.841301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.847679 (+ 6378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.847683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.848917 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.848928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.852694 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.852698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.852797 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.852819 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:53:20.852862 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:53:20.852926 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"thread_start_us":100,"threads_started":1}
I20260928 02:53:20.863375 718 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2c78c9da22433582c15e2f100fedee (DEFAULT_TABLE table=test-table [id=4719faf423114e90957424c382e3374a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:20.864621 718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2c78c9da22433582c15e2f100fedee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:20.869568 431 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2c78c9da22433582c15e2f100fedee (DEFAULT_TABLE table=test-table [id=4719faf423114e90957424c382e3374a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:20.870805 431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2c78c9da22433582c15e2f100fedee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:20.893877 820 tablet_bootstrap.cc:492] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: Bootstrap starting.
I20260928 02:53:20.896759 820 tablet_bootstrap.cc:654] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:20.897305 821 tablet_bootstrap.cc:492] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Bootstrap starting.
I20260928 02:53:20.900444 819 tablet_bootstrap.cc:492] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: Bootstrap starting.
I20260928 02:53:20.901916 820 log.cc:824] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:20.906016 821 tablet_bootstrap.cc:654] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:20.907099 821 log.cc:824] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:20.908222 819 tablet_bootstrap.cc:654] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:20.909336 819 log.cc:824] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:20.912510 820 tablet_bootstrap.cc:492] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: No bootstrap required, opened a new log
I20260928 02:53:20.912844 820 ts_tablet_manager.cc:1397] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260928 02:53:20.914593 821 tablet_bootstrap.cc:492] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: No bootstrap required, opened a new log
I20260928 02:53:20.914829 821 ts_tablet_manager.cc:1397] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260928 02:53:20.917351 820 raft_consensus.cc:348] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.917806 820 raft_consensus.cc:374] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:20.917904 820 raft_consensus.cc:729] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a681751e1e94032b78de71ed2bf6175, State: Initialized, Role: FOLLOWER
I20260928 02:53:20.918524 820 consensus_queue.cc:260] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.919364 821 raft_consensus.cc:348] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.919649 821 raft_consensus.cc:374] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:20.919720 821 raft_consensus.cc:729] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d64ca43a2c48ebaabf760e75222b47, State: Initialized, Role: FOLLOWER
I20260928 02:53:20.919907 820 ts_tablet_manager.cc:1428] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:53:20.920354 516 heartbeater.cc:503] Master 127.19.139.190:44483 was elected leader, sending a full tablet report...
I20260928 02:53:20.922212 819 tablet_bootstrap.cc:492] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: No bootstrap required, opened a new log
I20260928 02:53:20.922459 819 ts_tablet_manager.cc:1397] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260928 02:53:20.926265 821 consensus_queue.cc:260] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.927485 784 heartbeater.cc:503] Master 127.19.139.190:44483 was elected leader, sending a full tablet report...
I20260928 02:53:20.927556 821 ts_tablet_manager.cc:1428] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260928 02:53:20.928701 819 raft_consensus.cc:348] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.929020 819 raft_consensus.cc:374] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:20.929069 819 raft_consensus.cc:729] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cfdf89087749e2a2173471121082c0, State: Initialized, Role: FOLLOWER
I20260928 02:53:20.929572 819 consensus_queue.cc:260] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.930722 819 ts_tablet_manager.cc:1428] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260928 02:53:20.931092 650 heartbeater.cc:503] Master 127.19.139.190:44483 was elected leader, sending a full tablet report...
I20260928 02:53:20.957862 827 raft_consensus.cc:482] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:20.958112 827 raft_consensus.cc:504] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.959633 827 leader_election.cc:288] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a681751e1e94032b78de71ed2bf6175 (127.19.139.129:35809), e4d64ca43a2c48ebaabf760e75222b47 (127.19.139.131:36591)
W20260928 02:53:20.964365 517 tablet.cc:2369] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:20.966255 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.960273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:53789 (local address 127.19.139.129:35809)
0928 02:53:20.960423 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.960428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.961980 (+ 1552us) server_negotiation.cc:407] Connection header received
0928 02:53:20.962129 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.962133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.962186 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.962260 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:20.962664 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.963466 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.964953 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.965554 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.965648 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.965704 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:20.966042 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:53:20.966126 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:53:20.966668 828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.960078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:35809 (local address 127.19.139.130:53789)
0928 02:53:20.961885 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.961901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.962028 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.962390 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.962393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.962407 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:20.962545 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.962552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.963593 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.963596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.964815 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.964825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.965787 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.965790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.965904 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.965928 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:53:20.966427 (+ 499us) client_negotiation.cc:240] Negotiation successful
0928 02:53:20.966506 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1731,"thread_start_us":619,"threads_started":1}
I20260928 02:53:20.966954 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.960500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:37337 (local address 127.19.139.131:36591)
0928 02:53:20.960612 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.960616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.962296 (+ 1680us) server_negotiation.cc:407] Connection header received
0928 02:53:20.962469 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.962472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.962520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.962563 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:20.962936 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.963631 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.965763 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.966314 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.966400 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.966446 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:20.966748 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:53:20.966844 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:53:20.967296 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.960382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:36591 (local address 127.19.139.130:37337)
0928 02:53:20.962222 (+ 1840us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.962234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.962359 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.962681 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.962683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.962698 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:20.962812 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.962818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.963741 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.963744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.965632 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.965640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.966528 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.966532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.966623 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.966641 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:53:20.967093 (+ 452us) client_negotiation.cc:240] Negotiation successful
0928 02:53:20.967168 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1783,"thread_start_us":131,"threads_started":1}
I20260928 02:53:20.967669 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c78c9da22433582c15e2f100fedee" candidate_uuid: "a3cfdf89087749e2a2173471121082c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a681751e1e94032b78de71ed2bf6175" is_pre_election: true
I20260928 02:53:20.968008 470 raft_consensus.cc:2454] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cfdf89087749e2a2173471121082c0 in term 0.
I20260928 02:53:20.968557 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c78c9da22433582c15e2f100fedee" candidate_uuid: "a3cfdf89087749e2a2173471121082c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d64ca43a2c48ebaabf760e75222b47" is_pre_election: true
I20260928 02:53:20.968863 738 raft_consensus.cc:2454] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cfdf89087749e2a2173471121082c0 in term 0.
I20260928 02:53:20.969492 540 leader_election.cc:302] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cfdf89087749e2a2173471121082c0, e4d64ca43a2c48ebaabf760e75222b47; no voters:
I20260928 02:53:20.969836 827 raft_consensus.cc:2781] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:20.969935 827 raft_consensus.cc:482] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:20.969973 827 raft_consensus.cc:3037] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:20.971289 827 raft_consensus.cc:504] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.971673 827 leader_election.cc:288] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [CANDIDATE]: Term 1 election: Requested vote from peers 4a681751e1e94032b78de71ed2bf6175 (127.19.139.129:35809), e4d64ca43a2c48ebaabf760e75222b47 (127.19.139.131:36591)
I20260928 02:53:20.972584 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c78c9da22433582c15e2f100fedee" candidate_uuid: "a3cfdf89087749e2a2173471121082c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d64ca43a2c48ebaabf760e75222b47"
W20260928 02:53:20.972747 651 tablet.cc:2369] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:20.972725 738 raft_consensus.cc:3037] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:20.974035 738 raft_consensus.cc:2454] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cfdf89087749e2a2173471121082c0 in term 1.
I20260928 02:53:20.974454 540 leader_election.cc:302] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cfdf89087749e2a2173471121082c0, e4d64ca43a2c48ebaabf760e75222b47; no voters:
I20260928 02:53:20.974786 470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2c78c9da22433582c15e2f100fedee" candidate_uuid: "a3cfdf89087749e2a2173471121082c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a681751e1e94032b78de71ed2bf6175"
I20260928 02:53:20.974922 470 raft_consensus.cc:3037] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:20.976311 470 raft_consensus.cc:2454] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cfdf89087749e2a2173471121082c0 in term 1.
I20260928 02:53:20.976948 827 raft_consensus.cc:2781] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:20.977233 827 raft_consensus.cc:686] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [term 1 LEADER]: Becoming Leader. State: Replica: a3cfdf89087749e2a2173471121082c0, State: Running, Role: LEADER
I20260928 02:53:20.977653 827 consensus_queue.cc:237] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } }
I20260928 02:53:20.981746 318 catalog_manager.cc:5495] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cfdf89087749e2a2173471121082c0 (127.19.139.130). New cstate: current_term: 1 leader_uuid: "a3cfdf89087749e2a2173471121082c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cfdf89087749e2a2173471121082c0" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 } health_report { overall_health: UNKNOWN } } }
W20260928 02:53:20.996672 785 tablet.cc:2369] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:21.012960 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:21.007938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.19.139.130:35673)
0928 02:53:21.008057 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:21.008061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:21.008075 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:53:21.008175 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:21.008178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:21.008247 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:21.008353 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:21.008643 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.009409 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.010784 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.011127 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.011260 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.011609 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:21.012242 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:21.012367 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:21.012753 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:53:21.012830 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:53:21.019917 738 raft_consensus.cc:1261] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 1 FOLLOWER]: Refusing update from remote peer a3cfdf89087749e2a2173471121082c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:21.020996 827 consensus_queue.cc:1035] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d64ca43a2c48ebaabf760e75222b47" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:21.029349 470 raft_consensus.cc:1261] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 1 FOLLOWER]: Refusing update from remote peer a3cfdf89087749e2a2173471121082c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:21.030256 827 consensus_queue.cc:1035] T 1c2c78c9da22433582c15e2f100fedee P a3cfdf89087749e2a2173471121082c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a681751e1e94032b78de71ed2bf6175" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:21.037124 833 mvcc.cc:205] Tried to move back new op lower bound from 7334150148168437760 to 7334150148008202240. Current Snapshot: MvccSnapshot[applied={T|T < 7334150148168437760}]
I20260928 02:53:21.043779 836 mvcc.cc:205] Tried to move back new op lower bound from 7334150148168437760 to 7334150148008202240. Current Snapshot: MvccSnapshot[applied={T|T < 7334150148168437760}]
I20260928 02:53:21.044950 834 mvcc.cc:205] Tried to move back new op lower bound from 7334150148168437760 to 7334150148008202240. Current Snapshot: MvccSnapshot[applied={T|T < 7334150148168437760}]
I20260928 02:53:21.051759 317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:21.051949 317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:21.054569 317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bf384197772445785307727aa875741: Sending DeleteTablet for 3 replicas of tablet 1c2c78c9da22433582c15e2f100fedee
I20260928 02:53:21.055568 431 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c78c9da22433582c15e2f100fedee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:21 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54418
I20260928 02:53:21.055625 718 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c78c9da22433582c15e2f100fedee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:21 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:54020
I20260928 02:53:21.056124 584 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c2c78c9da22433582c15e2f100fedee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:21 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:40250
I20260928 02:53:21.056476 842 tablet_replica.cc:330] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175: stopping tablet replica
I20260928 02:53:21.056831 842 raft_consensus.cc:2229] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:21.057190 842 raft_consensus.cc:2258] T 1c2c78c9da22433582c15e2f100fedee P 4a681751e1e94032b78de71ed2bf6175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:21.057595 843 tablet_replica.cc:330] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: stopping tablet replica
I20260928 02:53:21.057668 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 356
I20260928 02:53:21.059267 843 raft_consensus.cc:2229] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:21.059597 843 raft_consensus.cc:2258] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:21.061982 843 ts_tablet_manager.cc:1905] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:21.064916 843 ts_tablet_manager.cc:1918] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:21.065023 843 log.cc:1196] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/1c2c78c9da22433582c15e2f100fedee
I20260928 02:53:21.065364 843 ts_tablet_manager.cc:1939] T 1c2c78c9da22433582c15e2f100fedee P e4d64ca43a2c48ebaabf760e75222b47: Deleting consensus metadata
W20260928 02:53:21.066020 303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:35809 (error 108)
I20260928 02:53:21.066304 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 518
I20260928 02:53:21.067796 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:21.067423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:35809 (local address 127.0.0.1:54420)
0928 02:53:21.067618 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:21.067704 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:35809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260928 02:53:21.067967 303 catalog_manager.cc:4584] TS 4a681751e1e94032b78de71ed2bf6175 (127.19.139.129:35809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c2c78c9da22433582c15e2f100fedee: Network error: Client connection negotiation failed: client connection to 127.19.139.129:35809: connect: Connection refused (error 111)
I20260928 02:53:21.069096 305 catalog_manager.cc:4841] TS e4d64ca43a2c48ebaabf760e75222b47 (127.19.139.131:36591): tablet 1c2c78c9da22433582c15e2f100fedee (table test-table [id=4719faf423114e90957424c382e3374a]) successfully deleted
I20260928 02:53:21.073501 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 653
I20260928 02:53:21.080593 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:21.080319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35673 (local address 127.0.0.1:40266)
0928 02:53:21.080434 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:21.080491 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:53:21.080631 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 32753
I20260928 02:53:21.085574 20014 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:53:21.421190 20014 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:53:21.421312 20014 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:53:21.756755 20014 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:53:21.756 UTC [31820] LOG: received smart shutdown request
2026-09-28 02:53:21.760 UTC [31820] LOG: background worker "logical replication launcher" (PID 31829) exited with exit code 1
2026-09-28 02:53:21.761 UTC [31824] LOG: shutting down
2026-09-28 02:53:21.944 UTC [31820] LOG: database system is shut down
2026-09-28T02:53:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49255 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:22 dist-test-slave-5vc0 krb5kdc[855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:22 dist-test-slave-5vc0 krb5kdc[855](info): set up 2 sockets
Sep 28 02:53:22 dist-test-slave-5vc0 krb5kdc[855](info): commencing operation
krb5kdc: starting...
W20260928 02:53:23.000080 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:53:23.123399 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:46609
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:46609,127.19.139.189:33091,127.19.139.188:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:23.244673 868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:23.244974 868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:23.245018 868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:23.249526 868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:23.249601 868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:23.249622 868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:23.249637 868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:23.249653 868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:23.255169 868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:46609,127.19.139.189:33091,127.19.139.188:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:46609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:23.256325 868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:23.257416 868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:23.266245 876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:23.266245 874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:23.266279 873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:23.266621 868 server_base.cc:1034] running on GCE node
I20260928 02:53:23.267216 868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:23.268416 868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:23.269589 868 hybrid_clock.cc:648] HybridClock initialized: now 1790564003269566 us; error 39 us; skew 500 ppm
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:23.275501 868 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:23.276620 868 webserver.cc:460] Webserver started at http://127.19.139.190:33477/ using document root <none> and password file <none>
I20260928 02:53:23.277244 868 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:23.277318 868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:23.277518 868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:23.279678 868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "64b2dd1a949d4c5691e2bfbcc50c418d"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "0a2d950f3241b845709874ceb9b3d2d1"
server_key_iv: "c2c3de83170a1050edb46a2c2d10f549"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.280345 868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "64b2dd1a949d4c5691e2bfbcc50c418d"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "0a2d950f3241b845709874ceb9b3d2d1"
server_key_iv: "c2c3de83170a1050edb46a2c2d10f549"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.284446 868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:53:23.287092 883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:23.288501 868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:23.288625 868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "64b2dd1a949d4c5691e2bfbcc50c418d"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "0a2d950f3241b845709874ceb9b3d2d1"
server_key_iv: "c2c3de83170a1050edb46a2c2d10f549"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.288734 868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:23.315141 868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:23.318593 868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:23.318795 868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:23.326030 868 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:46609
I20260928 02:53:23.326306 935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:46609 every 8 connection(s)
I20260928 02:53:23.327088 868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:23.330094 936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:23.331219 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 868
I20260928 02:53:23.331373 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:53:23.331682 20014 external_mini_cluster.cc:1428] Setting key 2007bf25186b926f5ab25ee49399f8fb
I20260928 02:53:23.334314 936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.139.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.189@KRBTEST.COM" created.
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:23.350907 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.336905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.19.139.190:46609)
0928 02:53:23.337524 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.337542 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.337707 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:53:23.339112 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.339117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.339344 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.339519 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.340470 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.341422 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.342635 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.343200 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.343357 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.346069 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.346075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.346086 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.346109 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.347652 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.348260 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.348265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.348266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.348380 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.348725 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.348728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.348729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.349169 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.349273 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.349756 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.350037 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":64896,"thread_start_us":158,"threads_started":1}
I20260928 02:53:23.350904 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.336905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46609 (local address 127.0.0.1:41278)
0928 02:53:23.337522 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.337582 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.338737 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.339571 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.339577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.339927 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.340289 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.340308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.341527 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.341529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.342518 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.342528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.343351 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.343354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.343457 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.344146 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.344188 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.345915 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.347773 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.347783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.347796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.348116 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.348466 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.348469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.348472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.348618 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.349288 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.349296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.349472 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.349670 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.350133 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":93,"spinlock_wait_cycles":768,"thread_start_us":158,"threads_started":1}
I20260928 02:53:23.353485 936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:53:23.354230 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.353953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44710)
0928 02:53:23.354084 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.354162 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:53:23.354390 887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
I20260928 02:53:23.355254 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.355022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44722)
0928 02:53:23.355135 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.355189 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:53:23.355392 936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:33091: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab.
Entry for principal kudu/127.19.139.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab.
I20260928 02:53:23.359635 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:33091
--webserver_interface=127.19.139.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab
--principal=kudu/127.19.139.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:46609,127.19.139.189:33091,127.19.139.188:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:23.404645 936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } attempt: 1
I20260928 02:53:23.405519 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.405216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44730)
0928 02:53:23.405371 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.405439 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:53:23.406286 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.406083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44746)
0928 02:53:23.406174 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.406223 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:53:23.406416 936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:33091: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
W20260928 02:53:23.470564 943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:23.470862 943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:23.470904 943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:53:23.474591 936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } attempt: 2
W20260928 02:53:23.475145 943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:23.475219 943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:23.475244 943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:23.475275 943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:23.475294 943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:23.475533 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.475190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44750)
0928 02:53:23.475377 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.475449 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:53:23.476257 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.476029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44756)
0928 02:53:23.476117 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.476194 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:53:23.476367 936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:33091: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33091: connect: Connection refused (error 111)
I20260928 02:53:23.481545 943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:46609,127.19.139.189:33091,127.19.139.188:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab
--principal=kudu/127.19.139.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.139.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:23.482649 943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:23.483678 943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:23.492306 949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:23.492319 951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:23.492378 943 server_base.cc:1034] running on GCE node
W20260928 02:53:23.492319 948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:23.493292 943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:23.494660 943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:23.495832 943 hybrid_clock.cc:648] HybridClock initialized: now 1790564003495808 us; error 44 us; skew 500 ppm
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:23.498569 943 init.cc:377] Logged in from keytab as kudu/127.19.139.189@KRBTEST.COM (short username kudu)
I20260928 02:53:23.499493 943 webserver.cc:460] Webserver started at http://127.19.139.189:40763/ using document root <none> and password file <none>
I20260928 02:53:23.500029 943 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:23.500092 943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:23.500308 943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:23.502341 943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data/instance:
uuid: "fe499a2e2485470cbd4c42d40b7be7d8"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "8a913636d56fd1b6df7e5715def57634"
server_key_iv: "1af7c57dd769514c8efdb8a6d3a708c1"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.502894 943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal/instance:
uuid: "fe499a2e2485470cbd4c42d40b7be7d8"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "8a913636d56fd1b6df7e5715def57634"
server_key_iv: "1af7c57dd769514c8efdb8a6d3a708c1"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.506783 943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:53:23.509451 958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:23.510813 943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:53:23.510965 943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal
uuid: "fe499a2e2485470cbd4c42d40b7be7d8"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "8a913636d56fd1b6df7e5715def57634"
server_key_iv: "1af7c57dd769514c8efdb8a6d3a708c1"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.511080 943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:23.528368 943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:23.531392 943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:23.531584 943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:23.538545 943 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.189:33091
I20260928 02:53:23.538802 1010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.189:33091 every 8 connection(s)
I20260928 02:53:23.539582 943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
I20260928 02:53:23.542531 1011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:23.546289 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 943
I20260928 02:53:23.546437 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-1/wal/instance
I20260928 02:53:23.546741 20014 external_mini_cluster.cc:1428] Setting key a0bb1c1cff45fb9cf5547d3ff4df5c1e
I20260928 02:53:23.546744 1011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.139.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.188@KRBTEST.COM" created.
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:23.562234 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.549337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.19.139.190:46609)
0928 02:53:23.549469 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.549473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.550658 (+ 1185us) server_negotiation.cc:407] Connection header received
0928 02:53:23.551657 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.551661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.551720 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.551794 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.553046 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.553573 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.554725 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.554977 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.555067 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.558405 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.558409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.558410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.558440 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.559935 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.560579 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.560583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.560584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.560668 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.561028 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.561031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.561032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.561226 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.561319 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.561914 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.562056 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:53:23.563124 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.549948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46609 (local address 127.0.0.1:41282)
0928 02:53:23.550490 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.550536 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.551447 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.551973 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.551981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.552383 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.552892 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.552905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.553700 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.553703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.554603 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.554613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.555126 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.555130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.555302 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.555869 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.555886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.558161 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.560074 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.560081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.560100 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.560476 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.560756 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.560759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.560762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.560926 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.561329 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.561336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.561630 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.561815 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.562207 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":148,"threads_started":1}
I20260928 02:53:23.564729 1011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:53:23.567530 936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } attempt: 3
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab.
Entry for principal kudu/127.19.139.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab.
I20260928 02:53:23.574934 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.565296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44770)
0928 02:53:23.565433 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.565445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.565529 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.566083 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.566086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.566236 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.566369 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.566375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.567409 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.567411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.568089 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.568099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.568797 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.568800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.568962 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.569504 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.569519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.570930 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.572980 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.572984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.572986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.573221 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.573601 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.573604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.573606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.573731 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.574345 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.574350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.574445 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.574545 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.574689 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":44}
I20260928 02:53:23.575225 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.565375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.19.139.189:33091)
0928 02:53:23.565722 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.565735 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.565758 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:53:23.565796 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.565800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.565964 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.566084 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.566463 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.567277 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.568219 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.568699 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.568789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.571076 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.571089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.571106 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.571128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.572872 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.573384 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.573391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.573393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.573507 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.573830 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.573834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.573836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.574185 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.574341 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.574650 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.575073 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":255,"thread_start_us":149,"threads_started":1}
I20260928 02:53:23.575522 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:40011
--webserver_interface=127.19.139.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab
--principal=kudu/127.19.139.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:46609,127.19.139.189:33091,127.19.139.188:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:23.577677 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.568032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44780)
0928 02:53:23.568199 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.568224 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.568307 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.568767 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.568770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.568984 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.569105 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.569145 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.569985 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.569989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.571264 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.571275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.571851 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.571855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.571983 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.572379 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.572397 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.573747 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.576206 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.576210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.576212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.576431 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.576683 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.576686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.576688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.576758 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.577148 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.577151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.577245 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.577350 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.577511 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:53:23.577697 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.568190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.19.139.189:33091)
0928 02:53:23.568532 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.568540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.568553 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:53:23.568585 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.568587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.568628 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.568716 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.569253 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.569848 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.571398 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.571686 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.571817 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.573889 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.573893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.573896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.573918 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.576055 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.576518 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.576521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.576523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.576605 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.576885 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.576889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.576891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.577056 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.577136 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.577428 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.577530 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260928 02:53:23.579006 936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:53:23.579097 1011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:53:23.579722 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.579495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38262)
0928 02:53:23.579609 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.579661 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:53:23.579849 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.579529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38274)
0928 02:53:23.579693 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.579771 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 02:53:23.580094 961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
I20260928 02:53:23.580564 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.580322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38282)
0928 02:53:23.580443 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.580489 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 02:53:23.580771 936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:40011: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
I20260928 02:53:23.581108 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.580845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38292)
0928 02:53:23.580993 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.581039 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260928 02:53:23.581257 1011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:40011: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
I20260928 02:53:23.611958 936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } attempt: 1
I20260928 02:53:23.612926 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.612563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38294)
0928 02:53:23.612721 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.612787 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:53:23.613618 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.613409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38308)
0928 02:53:23.613514 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.613557 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:53:23.613736 936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:40011: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
I20260928 02:53:23.630524 1011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } attempt: 1
I20260928 02:53:23.631448 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.631112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38320)
0928 02:53:23.631273 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.631361 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:53:23.632221 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.631969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38322)
0928 02:53:23.632075 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.632124 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:53:23.632339 1011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:40011: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
I20260928 02:53:23.688891 936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } attempt: 2
I20260928 02:53:23.689689 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.689411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38326)
0928 02:53:23.689549 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.689611 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:53:23.690335 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.690136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38330)
0928 02:53:23.690235 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.690276 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:53:23.690451 936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:40011: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
W20260928 02:53:23.690335 1018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:23.690665 1018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:23.690734 1018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:23.694851 1018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:23.694916 1018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:23.694940 1018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:23.694964 1018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:23.694993 1018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:23.700448 1018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:46609,127.19.139.189:33091,127.19.139.188:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab
--principal=kudu/127.19.139.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.139.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:23.700481 1011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } attempt: 2
I20260928 02:53:23.701267 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.700984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38346)
0928 02:53:23.701127 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.701188 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:53:23.701761 1018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:23.701967 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.701753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38350)
0928 02:53:23.701854 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.701907 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:53:23.702075 1011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:40011: Network error: Client connection negotiation failed: client connection to 127.19.139.188:40011: connect: Connection refused (error 111)
I20260928 02:53:23.702862 1018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:23.711510 1024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:23.711665 1018 server_base.cc:1034] running on GCE node
W20260928 02:53:23.711553 1027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:23.711524 1025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:23.712524 1018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:23.713733 1018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:23.714972 1018 hybrid_clock.cc:648] HybridClock initialized: now 1790564003714946 us; error 39 us; skew 500 ppm
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:23.717908 1018 init.cc:377] Logged in from keytab as kudu/127.19.139.188@KRBTEST.COM (short username kudu)
I20260928 02:53:23.718825 1018 webserver.cc:460] Webserver started at http://127.19.139.188:42427/ using document root <none> and password file <none>
I20260928 02:53:23.719375 1018 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:23.719435 1018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:23.719611 1018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:23.721727 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data/instance:
uuid: "18ff68151db14ce29717847c684d3aa5"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "04d15b4ec525a4cda4c70a4fbc5e3b38"
server_key_iv: "e2d7071948864c680db34649d1c54d26"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.722272 1018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal/instance:
uuid: "18ff68151db14ce29717847c684d3aa5"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "04d15b4ec525a4cda4c70a4fbc5e3b38"
server_key_iv: "e2d7071948864c680db34649d1c54d26"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.726243 1018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:53:23.728965 1034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:23.730347 1018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:23.730470 1018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal
uuid: "18ff68151db14ce29717847c684d3aa5"
format_stamp: "Formatted at 2026-09-28 02:53:23 on dist-test-slave-5vc0"
server_key: "04d15b4ec525a4cda4c70a4fbc5e3b38"
server_key_iv: "e2d7071948864c680db34649d1c54d26"
server_key_version: "encryptionkey@0"
I20260928 02:53:23.730575 1018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:23.751878 1018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:23.755141 1018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:23.755302 1018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:23.762264 1018 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.188:40011
I20260928 02:53:23.762550 1086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.188:40011 every 8 connection(s)
I20260928 02:53:23.763302 1018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
I20260928 02:53:23.766139 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:23.770812 1087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:53:23.772778 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1018
I20260928 02:53:23.772967 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563845681472-20014-0/minicluster-data/master-2/wal/instance
I20260928 02:53:23.773248 20014 external_mini_cluster.cc:1428] Setting key 2efb7164ef0f8ee78eed206596741112
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:53:23.785784 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.772791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.19.139.190:46609)
0928 02:53:23.772958 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.772963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.773954 (+ 991us) server_negotiation.cc:407] Connection header received
0928 02:53:23.775036 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.775039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.775090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.775181 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.776246 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.777048 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.778410 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.778749 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.778950 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.781769 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.781774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.781777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.781812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.783360 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.783925 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.783929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.783930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.784021 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.784420 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.784424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.784426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.784659 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.784764 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.785452 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.785609 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:53:23.786818 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.773234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46609 (local address 127.0.0.1:41284)
0928 02:53:23.773782 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.773843 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.774828 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.775239 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.775245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.775716 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.776091 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.776105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.777148 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.777151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.778283 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.778293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.778960 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.778963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.779104 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.779795 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.779817 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.781559 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.783472 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.783479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.783492 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.783793 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.784123 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.784126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.784128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.784325 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.784786 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.784792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.785141 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.785329 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.785691 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260928 02:53:23.786825 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.774293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.19.139.188:40011)
0928 02:53:23.774578 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.774586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.774622 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:53:23.774832 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.774849 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.775069 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.775244 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.776101 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.777377 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.778472 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.779140 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.779245 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.782142 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.782149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.782166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.782193 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.784005 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.784540 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.784545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.784546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.784630 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.784904 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.784907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.784908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.785204 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.785404 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.785703 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.786335 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260928 02:53:23.788395 1087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:53:23.789588 936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } attempt: 3
I20260928 02:53:23.793341 1011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } attempt: 3
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:23.798666 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.789033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33091 (local address 127.0.0.1:44786)
0928 02:53:23.789198 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.789209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.789275 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.789773 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.789776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.789976 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.790110 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.790118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.790972 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.790976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.791823 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.791831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.792580 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.792584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.792682 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.793203 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.793218 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.794599 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.796998 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.797001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.797003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.797289 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.797600 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.797603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.797605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.797688 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.798145 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.798148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.798248 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.798343 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.798482 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":3,"spinlock_wait_cycles":3456}
I20260928 02:53:23.799165 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.789936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38368)
0928 02:53:23.790031 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.790043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.790107 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.790577 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.790580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.790733 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.790808 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.790812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.791964 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.791967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.792630 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.792639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.793217 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.793221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.793300 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.793690 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.793704 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.795685 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.797634 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.797638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.797640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.797898 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.798178 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.798181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.798182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.798268 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.798699 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.798702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.798800 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.798888 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.799013 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:53:23.799211 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.789151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.19.139.189:33091)
0928 02:53:23.789271 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.789276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.789290 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:53:23.789358 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.789361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.789415 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.789495 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.790233 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.790862 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.791947 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.792371 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.792467 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.794786 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.794790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.794793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.794823 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.796568 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.797406 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.797410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.797412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.797501 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.797780 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.797783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.797785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.798015 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.798103 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.798940 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.799066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:53:23.799680 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.790254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.19.139.188:40011)
0928 02:53:23.790365 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.790369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.790380 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:53:23.790413 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.790416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.790451 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.790903 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.790981 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.791832 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.792733 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.793070 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.793172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.795847 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.795852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.795855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.795882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.797513 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.798014 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.798017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.798018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.798092 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.798394 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.798397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.798398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.798589 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.799357 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.799436 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.799542 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":48}
I20260928 02:53:23.799911 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.789513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.19.139.190:46609)
0928 02:53:23.789678 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.789681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.789696 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:53:23.789734 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.789737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.789777 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.789860 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.790633 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.791228 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.791990 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.792384 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.792450 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.796041 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.796045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.796047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.796068 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.797596 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.798297 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.798301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.798314 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.798399 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.798697 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.798701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.798702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.798904 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.799137 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.799644 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.799749 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":93}
I20260928 02:53:23.800318 1087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:53:23.803670 936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: Bootstrap starting.
I20260928 02:53:23.805685 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.794908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38378)
0928 02:53:23.795063 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.795078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.795149 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.795826 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.795829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.796755 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.796860 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.796866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.798340 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.798343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.799101 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.799111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.800692 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.800696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.800779 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.801213 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.801227 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.802551 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.804096 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.804099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.804101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.804366 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.804640 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.804643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.804645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.804744 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.805215 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.805218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.805317 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.805397 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.805528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:53:23.806217 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.795167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.19.139.188:40011)
0928 02:53:23.795577 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.795586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.795599 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:53:23.795631 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.795634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.795677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.795943 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.796986 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.798217 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.800267 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.800595 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.800688 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.802675 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.802678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.802680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.802700 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.803999 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.804462 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.804466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.804467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.804546 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.804862 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.804865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.804867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.805098 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.805855 (+ 757us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.805946 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.806058 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":768,"thread_start_us":151,"threads_started":1}
I20260928 02:53:23.806771 936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:23.807721 936 log.cc:824] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:23.809378 1011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: Bootstrap starting.
I20260928 02:53:23.809787 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.801063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.19.139.188:40011)
0928 02:53:23.801764 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.801768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.801782 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:53:23.801812 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.801815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.801871 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.801944 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.802348 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.803263 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.804443 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.804713 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.804794 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.806867 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.806870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.806873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.806897 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.808379 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.808730 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.808733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.808735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.808814 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.809040 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.809043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.809045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.809213 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.809291 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.809513 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.809648 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":57}
I20260928 02:53:23.809787 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.800827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:40011 (local address 127.0.0.1:38392)
0928 02:53:23.801236 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.801251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.801318 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.801982 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.801985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.802150 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.802239 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.802245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.803578 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.803581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.804334 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.804345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.804803 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.804806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.804917 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.805281 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.805294 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.806729 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:23.808475 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.808478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.808480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.808653 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.808892 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:23.808894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:23.808896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:23.808969 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:23.809294 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:23.809297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:23.809384 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:53:23.809464 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:53:23.809605 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":23}
I20260928 02:53:23.812491 936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: No bootstrap required, opened a new log
I20260928 02:53:23.812528 1011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:23.812973 1087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5: Bootstrap starting.
I20260928 02:53:23.813418 1011 log.cc:824] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:23.815836 1011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: No bootstrap required, opened a new log
I20260928 02:53:23.815869 1087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:23.816135 936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.816372 936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:23.816442 936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b2dd1a949d4c5691e2bfbcc50c418d, State: Initialized, Role: FOLLOWER
I20260928 02:53:23.816757 1087 log.cc:824] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:23.816886 936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.817747 1096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.817950 1096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.818246 936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:23.819020 1087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5: No bootstrap required, opened a new log
I20260928 02:53:23.819737 1011 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.819962 1011 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:23.820019 1011 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe499a2e2485470cbd4c42d40b7be7d8, State: Initialized, Role: FOLLOWER
W20260928 02:53:23.820428 889 tablet.cc:2369] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:53:23.823441 1107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:53:23.823624 1107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:53:23.824854 1011 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.826350 1011 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:23.823246 868 master_runner.cc:186] Error getting master registration for 127.19.139.189:33091: OK, instance_id { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" instance_seqno: 1790564003536477 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:53:23.826869 868 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:53:23.826681 1087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.827153 1087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:23.827248 1087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ff68151db14ce29717847c684d3aa5, State: Initialized, Role: FOLLOWER
I20260928 02:53:23.827785 1087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.828424 1115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.828572 1115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.829378 1087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:53:23 dist-test-slave-5vc0 krb5kdc[855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
W20260928 02:53:23.832582 1121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:53:23.832665 1121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 02:53:23.836233 1133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:53:23.836339 1133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:53:23.838868 1018 master_runner.cc:263] No leader master found from master 18ff68151db14ce29717847c684d3aa5
I20260928 02:53:23.838954 1018 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:53:23.840363 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.801494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.19.139.189:33091)
0928 02:53:23.801599 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.801603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.814273 (+ 12670us) server_negotiation.cc:407] Connection header received
0928 02:53:23.820542 (+ 6269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.820547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.820604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.820658 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.821666 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.822367 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.823405 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.823730 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.823798 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.831248 (+ 7450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.831253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.831256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.831287 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.833118 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.838649 (+ 5531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.838654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.838656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.838775 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.839135 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.839138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.839140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.839411 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.839597 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.840006 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.840184 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:53:23.844012 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:53:23.848891 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.849082 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.854168 943 master_runner.cc:263] No leader master found from master fe499a2e2485470cbd4c42d40b7be7d8
I20260928 02:53:23.854254 943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:53:23.873287 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.846760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.19.139.190:46609)
0928 02:53:23.846886 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.846891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.849384 (+ 2493us) server_negotiation.cc:407] Connection header received
0928 02:53:23.850961 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.850964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.851013 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.851062 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.853273 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.853990 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.859054 (+ 5064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.859416 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.859496 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.864863 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.864868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.864871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.864904 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.866799 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.869481 (+ 2682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.869486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.869488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.869591 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.871098 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.871102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.871103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.871369 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.871484 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.872958 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.873107 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:53:23.874008 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.847429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.19.139.189:33091)
0928 02:53:23.847540 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.847544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.849581 (+ 2037us) server_negotiation.cc:407] Connection header received
0928 02:53:23.849851 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.849854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.849906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.849957 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.855179 (+ 5222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.855915 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.860136 (+ 4221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.860500 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.860574 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.865244 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.865248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.865251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.865283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.867549 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.868814 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.868819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.868821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.868916 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.871806 (+ 2890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.871810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.871812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.872055 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.872189 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.873652 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.873787 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:53:23.878379 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.847958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.19.139.188:40011)
0928 02:53:23.848069 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.848073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.852099 (+ 4026us) server_negotiation.cc:407] Connection header received
0928 02:53:23.852303 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.852306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.852370 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.852422 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.857115 (+ 4693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.857886 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.862144 (+ 4258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.862508 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.862584 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.868448 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:23.868452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:23.868455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:23.868490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:23.870296 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.872505 (+ 2209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.872509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.872511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.872606 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:23.875729 (+ 3123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:23.875733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:23.875735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:23.876006 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:23.876139 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:23.878069 (+ 1930us) server_negotiation.cc:299] Negotiation successful
0928 02:53:23.878213 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:53:23.926563 1115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:23.926817 1115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.928125 1115 leader_election.cc:288] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b2dd1a949d4c5691e2bfbcc50c418d (127.19.139.190:46609), fe499a2e2485470cbd4c42d40b7be7d8 (127.19.139.189:33091)
I20260928 02:53:23.928704 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18ff68151db14ce29717847c684d3aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" is_pre_election: true
I20260928 02:53:23.932056 910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ff68151db14ce29717847c684d3aa5 in term 0.
I20260928 02:53:23.932754 1038 leader_election.cc:302] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ff68151db14ce29717847c684d3aa5, 64b2dd1a949d4c5691e2bfbcc50c418d; no voters:
I20260928 02:53:23.933140 1115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:23.933228 1115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:23.933271 1115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:23.934670 1115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.935081 1115 leader_election.cc:288] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [CANDIDATE]: Term 1 election: Requested vote from peers 64b2dd1a949d4c5691e2bfbcc50c418d (127.19.139.190:46609), fe499a2e2485470cbd4c42d40b7be7d8 (127.19.139.189:33091)
I20260928 02:53:23.935655 985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18ff68151db14ce29717847c684d3aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" is_pre_election: true
I20260928 02:53:23.936007 985 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ff68151db14ce29717847c684d3aa5 in term 0.
I20260928 02:53:23.937172 985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18ff68151db14ce29717847c684d3aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe499a2e2485470cbd4c42d40b7be7d8"
I20260928 02:53:23.937320 985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:23.938133 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18ff68151db14ce29717847c684d3aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d"
I20260928 02:53:23.938277 910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:23.939577 910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ff68151db14ce29717847c684d3aa5 in term 1.
I20260928 02:53:23.940822 985 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ff68151db14ce29717847c684d3aa5 in term 1.
I20260928 02:53:23.941208 1038 leader_election.cc:302] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ff68151db14ce29717847c684d3aa5, fe499a2e2485470cbd4c42d40b7be7d8; no voters:
I20260928 02:53:23.941592 1115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:23.941833 1115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 18ff68151db14ce29717847c684d3aa5, State: Running, Role: LEADER
I20260928 02:53:23.942217 1115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } }
I20260928 02:53:23.944252 1115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ff68151db14ce29717847c684d3aa5. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.946902 1115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:23.950168 1144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:23.951297 1144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:23.959911 985 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [term 1 FOLLOWER]: Refusing update from remote peer 18ff68151db14ce29717847c684d3aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:23.960704 1142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:23.964680 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ff68151db14ce29717847c684d3aa5. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.964833 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.965915 910 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [term 1 FOLLOWER]: Refusing update from remote peer 18ff68151db14ce29717847c684d3aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:23.966588 1142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:23.971093 1096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ff68151db14ce29717847c684d3aa5. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.971249 1096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.980264 1142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.980428 1142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:23.981920 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.982066 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.982632 1145 mvcc.cc:205] Tried to move back new op lower bound from 7334150160200052736 to 7334150160151056384. Current Snapshot: MvccSnapshot[applied={T|T < 7334150160200052736}]
I20260928 02:53:23.984674 1115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.984820 1115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:23.987879 1096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18ff68151db14ce29717847c684d3aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b2dd1a949d4c5691e2bfbcc50c418d" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46609 } } peers { permanent_uuid: "fe499a2e2485470cbd4c42d40b7be7d8" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33091 } } peers { permanent_uuid: "18ff68151db14ce29717847c684d3aa5" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 40011 } } }
I20260928 02:53:23.988029 1096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:53:23.988657 1144 catalog_manager.cc:1347] Generated new cluster ID: fb9fff0ccbe14f2181fc3ef40c390edc
I20260928 02:53:23.988734 1144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:23.999441 1148 mvcc.cc:205] Tried to move back new op lower bound from 7334150160200052736 to 7334150160151056384. Current Snapshot: MvccSnapshot[applied={T|T < 7334150160200052736}]
I20260928 02:53:24.021376 1144 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:24.022677 1144 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:24.037528 1144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18ff68151db14ce29717847c684d3aa5: Generated new TSK 0
I20260928 02:53:24.038569 1144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:53:24.825142 1107 catalog_manager.cc:1259] Loaded cluster ID: fb9fff0ccbe14f2181fc3ef40c390edc
I20260928 02:53:24.825238 1107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: loading cluster ID for follower catalog manager: success
I20260928 02:53:24.827092 1107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: acquiring CA information for follower catalog manager: success
I20260928 02:53:24.828406 1107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64b2dd1a949d4c5691e2bfbcc50c418d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:53:24.833925 1121 catalog_manager.cc:1259] Loaded cluster ID: fb9fff0ccbe14f2181fc3ef40c390edc
I20260928 02:53:24.834000 1121 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: loading cluster ID for follower catalog manager: success
I20260928 02:53:24.835726 1121 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: acquiring CA information for follower catalog manager: success
I20260928 02:53:24.836998 1121 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe499a2e2485470cbd4c42d40b7be7d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:53:25.107537 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 868
I20260928 02:53:25.112776 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 943
I20260928 02:53:25.117667 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1018
2026-09-28T02:53:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3077 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:25 dist-test-slave-5vc0 krb5kdc[1159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:25 dist-test-slave-5vc0 krb5kdc[1159](info): set up 2 sockets
Sep 28 02:53:25 dist-test-slave-5vc0 krb5kdc[1159](info): commencing operation
krb5kdc: starting...
W20260928 02:53:26.068126 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:53:26 dist-test-slave-5vc0 krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:53:26.191045 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:43265
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:43913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:43265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:26.301857 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:26.302171 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:26.302230 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:26.306420 1172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:26.306497 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:26.306524 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:26.306546 1172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:26.306574 1172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:26.312055 1172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:43913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:43265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:26.313150 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:26.314209 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:26.322706 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:26.322878 1172 server_base.cc:1034] running on GCE node
W20260928 02:53:26.322780 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:26.322706 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:26.323541 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:26.324753 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:26.325924 1172 hybrid_clock.cc:648] HybridClock initialized: now 1790564006325897 us; error 42 us; skew 500 ppm
Sep 28 02:53:26 dist-test-slave-5vc0 krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:26.328702 1172 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:26.329635 1172 webserver.cc:460] Webserver started at http://127.19.139.190:40451/ using document root <none> and password file <none>
I20260928 02:53:26.330181 1172 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:26.330245 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:26.330463 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:26.332629 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "5c6c94f43e704ca0a654ddbf679273e5"
format_stamp: "Formatted at 2026-09-28 02:53:26 on dist-test-slave-5vc0"
server_key: "2f3294f73f8c7b239ef0912a70ea0bdf"
server_key_iv: "5964e7c1ef3e427507c9909a61d7ef26"
server_key_version: "encryptionkey@0"
I20260928 02:53:26.333199 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "5c6c94f43e704ca0a654ddbf679273e5"
format_stamp: "Formatted at 2026-09-28 02:53:26 on dist-test-slave-5vc0"
server_key: "2f3294f73f8c7b239ef0912a70ea0bdf"
server_key_iv: "5964e7c1ef3e427507c9909a61d7ef26"
server_key_version: "encryptionkey@0"
I20260928 02:53:26.337224 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:53:26.339862 1187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:26.341190 1172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:53:26.341312 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "5c6c94f43e704ca0a654ddbf679273e5"
format_stamp: "Formatted at 2026-09-28 02:53:26 on dist-test-slave-5vc0"
server_key: "2f3294f73f8c7b239ef0912a70ea0bdf"
server_key_iv: "5964e7c1ef3e427507c9909a61d7ef26"
server_key_version: "encryptionkey@0"
I20260928 02:53:26.341442 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:26.353574 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:26.356987 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:26.357197 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:26.364435 1172 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:43265
I20260928 02:53:26.364521 1239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:43265 every 8 connection(s)
I20260928 02:53:26.365495 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:26.367514 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1172
I20260928 02:53:26.367632 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:53:26.367959 20014 external_mini_cluster.cc:1428] Setting key 0518bedd15a65109b4dabb005ac021f5
I20260928 02:53:26.368759 1240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:26.375967 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5: Bootstrap starting.
Sep 28 02:53:26 dist-test-slave-5vc0 krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:26.378690 1240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:26.379433 1240 log.cc:824] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:26.381467 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5: No bootstrap required, opened a new log
I20260928 02:53:26.383152 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:26.369427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.19.139.190:43265)
0928 02:53:26.369890 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:26.369909 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:26.369955 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:53:26.370800 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:26.370837 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:26.371291 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:26.371693 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:26.372434 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.373933 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.374792 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.375518 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.375689 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.378301 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:26.378307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:26.378325 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:26.378362 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:26.380286 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:26.380736 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:26.380742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:26.380747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:26.380865 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:26.381138 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:26.381141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:26.381143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:26.381569 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:26.381750 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:26.382054 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:53:26.382369 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":126,"threads_started":1}
I20260928 02:53:26.385242 1240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c94f43e704ca0a654ddbf679273e5" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43265 } }
I20260928 02:53:26.385531 1240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:26.385604 1240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6c94f43e704ca0a654ddbf679273e5, State: Initialized, Role: FOLLOWER
I20260928 02:53:26.386034 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:53:26.386086 1240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c94f43e704ca0a654ddbf679273e5" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43265 } }
I20260928 02:53:26.386251 1240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:26.386317 1240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:26.386427 1240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:26.387603 1240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c94f43e704ca0a654ddbf679273e5" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43265 } }
I20260928 02:53:26.387955 1240 leader_election.cc:302] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c6c94f43e704ca0a654ddbf679273e5; no voters:
I20260928 02:53:26.388278 1240 leader_election.cc:288] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:26.388382 1245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:26.388593 1245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5c6c94f43e704ca0a654ddbf679273e5, State: Running, Role: LEADER
I20260928 02:53:26.389184 1245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c94f43e704ca0a654ddbf679273e5" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43265 } }
I20260928 02:53:26.389497 1240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:26.390965 1247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c6c94f43e704ca0a654ddbf679273e5. Latest consensus state: current_term: 1 leader_uuid: "5c6c94f43e704ca0a654ddbf679273e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c94f43e704ca0a654ddbf679273e5" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43265 } } }
I20260928 02:53:26.391116 1247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:26.391001 1246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c6c94f43e704ca0a654ddbf679273e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6c94f43e704ca0a654ddbf679273e5" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43265 } } }
I20260928 02:53:26.391345 1246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:26.391445 1259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:26.394388 1259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:26.399610 1259 catalog_manager.cc:1347] Generated new cluster ID: 5f7c52509e034185a684b247f257a59f
I20260928 02:53:26.399688 1259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:26.410565 1259 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:26.411913 1259 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:26.418123 1259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c6c94f43e704ca0a654ddbf679273e5: Generated new TSK 0
I20260928 02:53:26.419076 1259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:53:26.442135 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1172
2026-09-28T02:53:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1322 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:26 dist-test-slave-5vc0 krb5kdc[1267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:26 dist-test-slave-5vc0 krb5kdc[1267](info): set up 2 sockets
Sep 28 02:53:26 dist-test-slave-5vc0 krb5kdc[1267](info): commencing operation
krb5kdc: starting...
W20260928 02:53:27.400312 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:53:27 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:53:27.520838 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41605
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:27.630966 1280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:27.631271 1280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:27.631340 1280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:27.635461 1280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:27.635537 1280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:27.635566 1280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:27.635587 1280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:27.635607 1280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:27.641083 1280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:27.642205 1280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:27.643272 1280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:27.651813 1288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:27.651861 1286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:27.651834 1285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:27.652069 1280 server_base.cc:1034] running on GCE node
I20260928 02:53:27.652884 1280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:27.654042 1280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:27.655233 1280 hybrid_clock.cc:648] HybridClock initialized: now 1790564007655208 us; error 36 us; skew 500 ppm
Sep 28 02:53:27 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564007, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:27.658080 1280 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:27.658979 1280 webserver.cc:460] Webserver started at http://127.19.139.190:41327/ using document root <none> and password file <none>
I20260928 02:53:27.659495 1280 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:27.659554 1280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:27.659718 1280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:27.661835 1280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "927572b566e847bfab871367885bbdef"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "726a0e6532743acd8abce3ba548cd3e9"
server_key_iv: "83af73dc518c10a09523003bd360a8d0"
server_key_version: "encryptionkey@0"
I20260928 02:53:27.662370 1280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "927572b566e847bfab871367885bbdef"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "726a0e6532743acd8abce3ba548cd3e9"
server_key_iv: "83af73dc518c10a09523003bd360a8d0"
server_key_version: "encryptionkey@0"
I20260928 02:53:27.666134 1280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:53:27.668520 1295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:27.669709 1280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:53:27.669857 1280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "927572b566e847bfab871367885bbdef"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "726a0e6532743acd8abce3ba548cd3e9"
server_key_iv: "83af73dc518c10a09523003bd360a8d0"
server_key_version: "encryptionkey@0"
I20260928 02:53:27.669943 1280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:27.678994 1280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:27.682173 1280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:27.682339 1280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:27.689047 1280 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:41605
I20260928 02:53:27.689303 1347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:41605 every 8 connection(s)
I20260928 02:53:27.690021 1280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:27.692853 1348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:27.696946 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1280
I20260928 02:53:27.697062 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:53:27.697329 20014 external_mini_cluster.cc:1428] Setting key 5840244f185e10e7a096c9907ea6f9c3
I20260928 02:53:27.698851 1348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap starting.
I20260928 02:53:27.701678 1348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:27.702626 1348 log.cc:824] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:27.704835 1348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: No bootstrap required, opened a new log
Sep 28 02:53:27 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:27.709316 1348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:27.709628 1348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:27.709692 1348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927572b566e847bfab871367885bbdef, State: Initialized, Role: FOLLOWER
I20260928 02:53:27.710217 1348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:27.710381 1348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:27.710492 1348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:27.710590 1348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:27.711521 1348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:27.711906 1348 leader_election.cc:302] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 927572b566e847bfab871367885bbdef; no voters:
I20260928 02:53:27.712255 1348 leader_election.cc:288] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:27.712363 1353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:27.712576 1353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 LEADER]: Becoming Leader. State: Replica: 927572b566e847bfab871367885bbdef, State: Running, Role: LEADER
I20260928 02:53:27.712775 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:27.698743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.19.139.190:41605)
0928 02:53:27.699175 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:27.699194 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:27.699239 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:53:27.700121 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:27.700148 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:27.700668 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:27.701050 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:27.701732 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.703193 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.704100 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.704638 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.704786 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.707507 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:27.707514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:27.707536 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:27.707575 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:27.709504 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:27.710018 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:27.710025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:27.710031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:27.710174 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:27.710448 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:27.710452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:27.710453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:27.710965 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:27.711163 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:27.711451 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:53:27.711771 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260928 02:53:27.712967 1353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:27.713526 1348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:27.714166 1354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927572b566e847bfab871367885bbdef. Latest consensus state: current_term: 1 leader_uuid: "927572b566e847bfab871367885bbdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } } }
I20260928 02:53:27.714339 1354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:27.715509 1355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "927572b566e847bfab871367885bbdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } } }
I20260928 02:53:27.715610 1355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: This master's current role is: LEADER
W20260928 02:53:27.719043 1368 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:53:27.719146 1368 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:53:27.719259 1362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:27.720093 1362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:27.725574 1362 catalog_manager.cc:1347] Generated new cluster ID: e7b633dea144431995061677ff356006
I20260928 02:53:27.725746 1362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:27.734385 1362 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:27.735374 1362 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:27.751438 1362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Generated new TSK 0
I20260928 02:53:27.752213 1362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:53:27.783604 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41605
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:27.897509 1374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:27.897807 1374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:27.897871 1374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:27.901949 1374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:27.902021 1374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:27.902105 1374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:53:27.907578 1374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:27.908713 1374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:27.909752 1374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:27.919045 1380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:27.919055 1382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:27.919070 1379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:27.919342 1374 server_base.cc:1034] running on GCE node
I20260928 02:53:27.919732 1374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:27.920429 1374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:27.921598 1374 hybrid_clock.cc:648] HybridClock initialized: now 1790564007921580 us; error 35 us; skew 500 ppm
Sep 28 02:53:27 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564007, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:27.924234 1374 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:53:27.925123 1374 webserver.cc:460] Webserver started at http://127.19.139.129:34567/ using document root <none> and password file <none>
I20260928 02:53:27.925670 1374 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:27.925737 1374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:27.925957 1374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:27.928071 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "b41748e7d1c747c891ba9579efae0377"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "06f5e3cfe879f28b6d33b0ef4aeeed5e"
server_key_iv: "e0e75fe9de8b1629619ffa215658bb1c"
server_key_version: "encryptionkey@0"
I20260928 02:53:27.928681 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "b41748e7d1c747c891ba9579efae0377"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "06f5e3cfe879f28b6d33b0ef4aeeed5e"
server_key_iv: "e0e75fe9de8b1629619ffa215658bb1c"
server_key_version: "encryptionkey@0"
I20260928 02:53:27.932557 1374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:53:27.935145 1389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:27.936535 1374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:27.936658 1374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "b41748e7d1c747c891ba9579efae0377"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "06f5e3cfe879f28b6d33b0ef4aeeed5e"
server_key_iv: "e0e75fe9de8b1629619ffa215658bb1c"
server_key_version: "encryptionkey@0"
I20260928 02:53:27.936760 1374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:27.957648 1374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:27.961102 1374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:27.961282 1374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:27.961833 1374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:27.962644 1374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:27.962692 1374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:27.962731 1374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:27.962745 1374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:27.971042 1374 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:43699
I20260928 02:53:27.971094 1502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:43699 every 8 connection(s)
I20260928 02:53:27.971935 1374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:27.980566 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1374
I20260928 02:53:27.980715 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:27.981021 20014 external_mini_cluster.cc:1428] Setting key 2cdfc9e5c253d8a147199ac560c4c774
Sep 28 02:53:27 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564007, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:27.986152 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:27.973766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:46121 (local address 127.19.139.190:41605)
0928 02:53:27.973897 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:27.973901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:27.974801 (+ 900us) server_negotiation.cc:407] Connection header received
0928 02:53:27.976244 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:27.976248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:27.976308 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:27.976403 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:27.977650 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.978180 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.979323 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.979561 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.979650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.982524 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:27.982528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:27.982530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:27.982557 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:27.984029 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:27.984556 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:27.984560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:27.984562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:27.984664 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:27.985036 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:27.985039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:27.985041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:27.985270 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:27.985367 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:27.985864 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 02:53:27.985990 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:53:27.986935 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:27.974211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:46121)
0928 02:53:27.974652 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:27.974696 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:27.975985 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:27.976601 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:27.976609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:27.977005 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:27.977495 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.977509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.978302 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.978304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.979204 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.979214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.979708 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.979712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.979882 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.980493 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:27.980515 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:27.982357 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:27.984144 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:27.984149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:27.984193 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:27.984472 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:27.984778 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:27.984781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:27.984782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:27.984939 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:27.985367 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:27.985372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:27.985618 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:53:27.985788 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:53:27.986054 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260928 02:53:27.988231 1503 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41605
I20260928 02:53:27.988508 1503 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:27.989110 1503 heartbeater.cc:511] Master 127.19.139.190:41605 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:53:27.990779 1312 ts_manager.cc:194] Registered new tserver with Master: b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699)
I20260928 02:53:27.992029 1312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:46121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:53:28.007117 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41605
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:28.116972 1508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:28.117250 1508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:28.117295 1508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:28.121389 1508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:28.121464 1508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:28.121526 1508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:53:28.126989 1508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:28.128089 1508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:28.129269 1508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:28.139595 1513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:28.139763 1508 server_base.cc:1034] running on GCE node
W20260928 02:53:28.139657 1514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:28.139580 1516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:28.140347 1508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:28.141093 1508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:28.142248 1508 hybrid_clock.cc:648] HybridClock initialized: now 1790564008142235 us; error 31 us; skew 500 ppm
Sep 28 02:53:28 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564008, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:28.145145 1508 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:53:28.146130 1508 webserver.cc:460] Webserver started at http://127.19.139.130:43939/ using document root <none> and password file <none>
I20260928 02:53:28.146757 1508 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:28.146828 1508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:28.147029 1508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:28.149215 1508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "5960f06594b9411896209f57cafddc71"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "4243f9965cddef8188fc9f7de1c51f80"
server_key_iv: "92fdfe15bdb606e8ae207dda6a0edfd3"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.149793 1508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "5960f06594b9411896209f57cafddc71"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "4243f9965cddef8188fc9f7de1c51f80"
server_key_iv: "92fdfe15bdb606e8ae207dda6a0edfd3"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.153738 1508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:53:28.156370 1523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.157653 1508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:53:28.157771 1508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "5960f06594b9411896209f57cafddc71"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "4243f9965cddef8188fc9f7de1c51f80"
server_key_iv: "92fdfe15bdb606e8ae207dda6a0edfd3"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.157881 1508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:28.181602 1508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:28.184978 1508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:28.185165 1508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:28.185786 1508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:28.186657 1508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:28.186707 1508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.186759 1508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:28.186785 1508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.195578 1508 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:35097
I20260928 02:53:28.195641 1636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:35097 every 8 connection(s)
I20260928 02:53:28.196596 1508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:28.203579 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1508
I20260928 02:53:28.203723 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:28.204011 20014 external_mini_cluster.cc:1428] Setting key 6869d3bc76f7c5aba2d6b557cbef35aa
Sep 28 02:53:28 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564008, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:28.211992 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.198452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:59323 (local address 127.19.139.190:41605)
0928 02:53:28.198621 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.198626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.199529 (+ 903us) server_negotiation.cc:407] Connection header received
0928 02:53:28.201037 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.201041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.201095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.201172 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.202570 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.203159 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.204631 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.204937 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.205022 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.208079 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:28.208083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:28.208085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:28.208112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:28.209452 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:28.210137 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:28.210140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:28.210141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:28.210235 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:28.210639 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:28.210641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:28.210642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:28.210858 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:28.210938 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.211636 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.211813 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:53:28.213186 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.198856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:59323)
0928 02:53:28.199357 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.199405 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.200816 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.201385 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.201393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.201871 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:28.202404 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.202418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.203303 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.203306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.204495 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.204508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.205099 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.205103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.205308 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.206030 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:28.206050 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:28.207858 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:28.209590 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:28.209597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:28.209618 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:28.210000 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:28.210336 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:28.210340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:28.210342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:28.210550 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:28.210959 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:28.210967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:28.211359 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.211586 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.211982 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:53:28.214713 1637 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41605
I20260928 02:53:28.215051 1637 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:28.215638 1637 heartbeater.cc:511] Master 127.19.139.190:41605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:53:28.216836 1312 ts_manager.cc:194] Registered new tserver with Master: 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
I20260928 02:53:28.217397 1312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:59323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:53:28.231518 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41605
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:28.343391 1642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:28.343695 1642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:28.343739 1642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:28.348038 1642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:28.348112 1642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:28.348210 1642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:53:28.353696 1642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:28.354822 1642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:28.355877 1642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:28.366417 1648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:28.366429 1647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:28.366580 1642 server_base.cc:1034] running on GCE node
W20260928 02:53:28.366427 1650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:28.367210 1642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:28.367869 1642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:28.369053 1642 hybrid_clock.cc:648] HybridClock initialized: now 1790564008369010 us; error 58 us; skew 500 ppm
Sep 28 02:53:28 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564008, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:28.376560 1642 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:53:28.377641 1642 webserver.cc:460] Webserver started at http://127.19.139.131:43715/ using document root <none> and password file <none>
I20260928 02:53:28.378235 1642 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:28.378309 1642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:28.378535 1642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:28.380928 1642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "b25e2e97042b4a7e88236e2af97d41c8"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "c30d02390d04379e5ace8b56321b0d2d"
server_key_iv: "a5ac1a80186d82e6a726f7ee0dc79de5"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.381503 1642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "b25e2e97042b4a7e88236e2af97d41c8"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "c30d02390d04379e5ace8b56321b0d2d"
server_key_iv: "a5ac1a80186d82e6a726f7ee0dc79de5"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.385406 1642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:53:28.387904 1657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.389221 1642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:53:28.389345 1642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "b25e2e97042b4a7e88236e2af97d41c8"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "c30d02390d04379e5ace8b56321b0d2d"
server_key_iv: "a5ac1a80186d82e6a726f7ee0dc79de5"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.389446 1642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:28.412364 1642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:28.415812 1642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:28.415997 1642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:28.416666 1642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:28.417554 1642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:28.417605 1642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.417660 1642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:28.417688 1642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.426270 1642 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:38109
I20260928 02:53:28.426298 1770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:38109 every 8 connection(s)
I20260928 02:53:28.427172 1642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:28.428064 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1642
I20260928 02:53:28.428220 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:28.428503 20014 external_mini_cluster.cc:1428] Setting key e9272813272e1db470e4a17c18312707
Sep 28 02:53:28 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564008, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:28.440984 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.429263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:56453 (local address 127.19.139.190:41605)
0928 02:53:28.429411 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.429414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.430457 (+ 1043us) server_negotiation.cc:407] Connection header received
0928 02:53:28.431843 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.431847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.431893 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.431967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.433276 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.433814 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.435000 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.435221 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.435302 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.437673 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:28.437677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:28.437679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:28.437706 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:28.438962 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:28.439493 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:28.439495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:28.439497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:28.439568 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:28.439871 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:28.439873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:28.439874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:28.440056 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:28.440130 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.440690 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.440801 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:53:28.441881 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.429785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:56453)
0928 02:53:28.430299 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.430354 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.431617 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.432105 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.432113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.432632 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:28.433133 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.433147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.433921 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.433924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.434872 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.434882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.435343 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.435345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.435497 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.436007 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:28.436023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:28.437512 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:28.439091 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:28.439096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:28.439110 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:28.439407 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:28.439632 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:28.439635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:28.439636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:28.439778 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:28.440136 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:28.440141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:28.440466 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.440634 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.440923 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260928 02:53:28.442948 1771 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41605
I20260928 02:53:28.443178 1771 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:28.443648 1771 heartbeater.cc:511] Master 127.19.139.190:41605 requested a full tablet report, sending...
I20260928 02:53:28.444693 1312 ts_manager.cc:194] Registered new tserver with Master: b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109)
I20260928 02:53:28.445255 1312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:56453
I20260928 02:53:28.454803 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:53:28.464280 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.456825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.19.139.190:41605)
0928 02:53:28.456967 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.456972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.457118 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:53:28.457247 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.457250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.457294 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.457366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.458125 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.458653 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.459487 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.459704 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.459792 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.461303 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:28.461306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:28.461308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:28.461342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:28.462661 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:28.463070 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:28.463074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:28.463075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:28.463150 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:28.463390 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:28.463394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:28.463396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:28.463623 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:28.463747 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.463975 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.464087 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:53:28.467103 1312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:28.470211 1312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:28.485184 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.480278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.19.139.130:35097)
0928 02:53:28.480897 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.480908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.480931 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:53:28.481000 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.481004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.481189 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.481323 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:28.481491 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.482226 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.483852 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.484534 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.484650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.484811 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.484937 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.485030 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":161,"threads_started":1}
I20260928 02:53:28.485180 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.479922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35097 (local address 127.0.0.1:59916)
0928 02:53:28.480558 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.480572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.480658 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.481304 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.481307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.481332 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:28.481407 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.481413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.482357 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.482361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.483716 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.483726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.484640 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.484644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.484751 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.484789 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.484900 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.485028 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":101,"threads_started":1}
I20260928 02:53:28.485879 1785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.479453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.19.139.131:38109)
0928 02:53:28.479943 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.479956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.480380 (+ 424us) server_negotiation.cc:407] Connection header received
0928 02:53:28.480586 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.480592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.480810 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.480942 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:28.481357 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.482478 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.483926 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.484882 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.485025 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.485159 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.485680 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.485774 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":110,"threads_started":1}
I20260928 02:53:28.485889 1782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.479085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:38109 (local address 127.0.0.1:35266)
0928 02:53:28.480293 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.480304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.480436 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.481128 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.481130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.481145 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:28.481235 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.481241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.482637 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.482639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.483807 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.483818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.485224 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.485227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.485530 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.485553 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.485623 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.485691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":166,"thread_start_us":97,"threads_started":1}
I20260928 02:53:28.488518 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.479053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43699 (local address 127.0.0.1:37010)
0928 02:53:28.480212 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.480269 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.480410 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.480979 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.480985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.481010 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:28.481147 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.481153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.482609 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.482613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.483830 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.483838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.485292 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.485296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.485375 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.485395 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.488299 (+ 2904us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.488383 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":26,"thread_start_us":110,"threads_started":1}
I20260928 02:53:28.488590 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.479219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.19.139.129:43699)
0928 02:53:28.479943 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.479958 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.480554 (+ 596us) server_negotiation.cc:407] Connection header received
0928 02:53:28.480626 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.480630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.480810 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.480951 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:28.481273 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.482478 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.483951 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.485008 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.485115 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.485215 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.488346 (+ 3131us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.488456 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":124,"threads_started":1}
I20260928 02:53:28.488189 1571 tablet_service.cc:1467] Processing CreateTablet for tablet eba1172c8f8348cc97494de9add2c4b0 (DEFAULT_TABLE table=test-table [id=cb94e2c44e324362b3d7a02a2da05713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:28.488302 1705 tablet_service.cc:1467] Processing CreateTablet for tablet eba1172c8f8348cc97494de9add2c4b0 (DEFAULT_TABLE table=test-table [id=cb94e2c44e324362b3d7a02a2da05713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:28.489338 1705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba1172c8f8348cc97494de9add2c4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:28.489346 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba1172c8f8348cc97494de9add2c4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:28.491822 1437 tablet_service.cc:1467] Processing CreateTablet for tablet eba1172c8f8348cc97494de9add2c4b0 (DEFAULT_TABLE table=test-table [id=cb94e2c44e324362b3d7a02a2da05713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:28.493141 1437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba1172c8f8348cc97494de9add2c4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:28.496412 1788 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Bootstrap starting.
I20260928 02:53:28.497509 1789 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Bootstrap starting.
I20260928 02:53:28.498854 1788 tablet_bootstrap.cc:654] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:28.499600 1788 log.cc:824] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:28.499704 1789 tablet_bootstrap.cc:654] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:28.501010 1789 log.cc:824] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:28.501658 1791 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Bootstrap starting.
I20260928 02:53:28.501775 1788 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: No bootstrap required, opened a new log
I20260928 02:53:28.501978 1788 ts_tablet_manager.cc:1397] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:53:28.503993 1791 tablet_bootstrap.cc:654] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:28.504946 1791 log.cc:824] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:28.505867 1789 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: No bootstrap required, opened a new log
I20260928 02:53:28.505842 1788 raft_consensus.cc:348] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.506063 1789 ts_tablet_manager.cc:1397] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 02:53:28.506124 1788 raft_consensus.cc:374] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:28.506202 1788 raft_consensus.cc:729] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5960f06594b9411896209f57cafddc71, State: Initialized, Role: FOLLOWER
I20260928 02:53:28.506697 1788 consensus_queue.cc:260] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.507042 1791 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: No bootstrap required, opened a new log
I20260928 02:53:28.507287 1791 ts_tablet_manager.cc:1397] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:53:28.507475 1637 heartbeater.cc:503] Master 127.19.139.190:41605 was elected leader, sending a full tablet report...
I20260928 02:53:28.507802 1788 ts_tablet_manager.cc:1428] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:53:28.509548 1789 raft_consensus.cc:348] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.509773 1789 raft_consensus.cc:374] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:28.509853 1789 raft_consensus.cc:729] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25e2e97042b4a7e88236e2af97d41c8, State: Initialized, Role: FOLLOWER
I20260928 02:53:28.510268 1789 consensus_queue.cc:260] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.511101 1771 heartbeater.cc:503] Master 127.19.139.190:41605 was elected leader, sending a full tablet report...
I20260928 02:53:28.511292 1789 ts_tablet_manager.cc:1428] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:53:28.512796 1791 raft_consensus.cc:348] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.513058 1791 raft_consensus.cc:374] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:28.513121 1791 raft_consensus.cc:729] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41748e7d1c747c891ba9579efae0377, State: Initialized, Role: FOLLOWER
I20260928 02:53:28.513530 1791 consensus_queue.cc:260] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.514232 1503 heartbeater.cc:503] Master 127.19.139.190:41605 was elected leader, sending a full tablet report...
I20260928 02:53:28.514515 1791 ts_tablet_manager.cc:1428] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:53:28.551075 1794 raft_consensus.cc:482] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:28.551296 1794 raft_consensus.cc:504] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.552549 1794 leader_election.cc:288] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699), b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109)
I20260928 02:53:28.556306 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.552807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:38109 (local address 127.19.139.130:50379)
0928 02:53:28.552932 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.552947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.553044 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.553323 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.553327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.553355 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:28.553476 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.553483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.554265 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.554268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.555080 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.555111 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.555843 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.555847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.555957 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.555977 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.556042 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.556118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:53:28.556358 1785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.552919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:50379 (local address 127.19.139.131:38109)
0928 02:53:28.553047 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.553052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.553068 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:53:28.553123 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.553126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.553172 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.553253 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:28.553645 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.554151 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.555255 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.555724 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.555841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.555899 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.556072 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.556145 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:53:28.556761 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.552918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:39379 (local address 127.19.139.129:43699)
0928 02:53:28.553039 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.553043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.553586 (+ 543us) server_negotiation.cc:407] Connection header received
0928 02:53:28.553710 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.553713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.553753 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.553817 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:28.554078 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.554557 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.555554 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.556030 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.556134 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.556219 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.556574 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.556648 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:53:28.557106 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.552807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43699 (local address 127.19.139.130:39379)
0928 02:53:28.553482 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.553496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.553592 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.553858 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.553861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.553877 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:28.553978 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.553983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.554653 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.554656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.555437 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.555446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.556335 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.556340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.556442 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.556466 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:53:28.556906 (+ 440us) client_negotiation.cc:240] Negotiation successful
0928 02:53:28.556980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":301,"thread_start_us":133,"threads_started":1}
I20260928 02:53:28.557081 1725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba1172c8f8348cc97494de9add2c4b0" candidate_uuid: "5960f06594b9411896209f57cafddc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25e2e97042b4a7e88236e2af97d41c8" is_pre_election: true
I20260928 02:53:28.557415 1725 raft_consensus.cc:2454] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5960f06594b9411896209f57cafddc71 in term 0.
I20260928 02:53:28.557734 1457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba1172c8f8348cc97494de9add2c4b0" candidate_uuid: "5960f06594b9411896209f57cafddc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41748e7d1c747c891ba9579efae0377" is_pre_election: true
I20260928 02:53:28.557889 1527 leader_election.cc:302] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5960f06594b9411896209f57cafddc71, b25e2e97042b4a7e88236e2af97d41c8; no voters:
I20260928 02:53:28.558038 1457 raft_consensus.cc:2454] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5960f06594b9411896209f57cafddc71 in term 0.
I20260928 02:53:28.558171 1794 raft_consensus.cc:2781] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:28.558246 1794 raft_consensus.cc:482] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:28.558300 1794 raft_consensus.cc:3037] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:28.559360 1794 raft_consensus.cc:504] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.559684 1794 leader_election.cc:288] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [CANDIDATE]: Term 1 election: Requested vote from peers b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699), b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109)
I20260928 02:53:28.560014 1457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba1172c8f8348cc97494de9add2c4b0" candidate_uuid: "5960f06594b9411896209f57cafddc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41748e7d1c747c891ba9579efae0377"
I20260928 02:53:28.560052 1725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba1172c8f8348cc97494de9add2c4b0" candidate_uuid: "5960f06594b9411896209f57cafddc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25e2e97042b4a7e88236e2af97d41c8"
I20260928 02:53:28.560125 1457 raft_consensus.cc:3037] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:28.560191 1725 raft_consensus.cc:3037] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:28.561206 1457 raft_consensus.cc:2454] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5960f06594b9411896209f57cafddc71 in term 1.
I20260928 02:53:28.561303 1725 raft_consensus.cc:2454] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5960f06594b9411896209f57cafddc71 in term 1.
I20260928 02:53:28.561478 1526 leader_election.cc:302] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5960f06594b9411896209f57cafddc71, b41748e7d1c747c891ba9579efae0377; no voters:
I20260928 02:53:28.561640 1794 raft_consensus.cc:2781] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:28.561860 1794 raft_consensus.cc:686] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 1 LEADER]: Becoming Leader. State: Replica: 5960f06594b9411896209f57cafddc71, State: Running, Role: LEADER
I20260928 02:53:28.562188 1794 consensus_queue.cc:237] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:28.565265 1311 catalog_manager.cc:5495] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5960f06594b9411896209f57cafddc71 (127.19.139.130). New cstate: current_term: 1 leader_uuid: "5960f06594b9411896209f57cafddc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } health_report { overall_health: HEALTHY } } }
I20260928 02:53:28.584309 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.580911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.19.139.130:35097)
0928 02:53:28.581039 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.581043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.581057 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:53:28.581185 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.581187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.581244 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.581332 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:28.581540 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.582077 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.582894 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.583109 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.583187 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.583474 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:28.583916 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:28.584015 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:28.584044 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 02:53:28.584098 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:53:28.590317 1725 raft_consensus.cc:1261] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Refusing update from remote peer 5960f06594b9411896209f57cafddc71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:28.590315 1457 raft_consensus.cc:1261] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Refusing update from remote peer 5960f06594b9411896209f57cafddc71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:28.591058 1798 consensus_queue.cc:1035] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:28.591243 1794 consensus_queue.cc:1035] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:28.599941 1802 mvcc.cc:205] Tried to move back new op lower bound from 7334150179178635264 to 7334150179073482752. Current Snapshot: MvccSnapshot[applied={T|T < 7334150179178635264}]
I20260928 02:53:28.606225 1311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:28.606413 1311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:28.608726 1311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Sending DeleteTablet for 3 replicas of tablet eba1172c8f8348cc97494de9add2c4b0
I20260928 02:53:28.609570 1437 tablet_service.cc:1514] Processing DeleteTablet for tablet eba1172c8f8348cc97494de9add2c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:28 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:37010
I20260928 02:53:28.609797 1705 tablet_service.cc:1514] Processing DeleteTablet for tablet eba1172c8f8348cc97494de9add2c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:28 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:35266
I20260928 02:53:28.609906 1571 tablet_service.cc:1514] Processing DeleteTablet for tablet eba1172c8f8348cc97494de9add2c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:28 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:59916
I20260928 02:53:28.610307 1811 tablet_replica.cc:330] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: stopping tablet replica
I20260928 02:53:28.610414 1812 tablet_replica.cc:330] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: stopping tablet replica
I20260928 02:53:28.610484 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1374
I20260928 02:53:28.610730 1811 raft_consensus.cc:2229] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:28.610741 1812 raft_consensus.cc:2229] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:28.611157 1812 raft_consensus.cc:2258] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:28.611521 1811 raft_consensus.cc:2258] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:28.613030 1812 ts_tablet_manager.cc:1905] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:28.613030 1811 ts_tablet_manager.cc:1905] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:28.615978 1811 ts_tablet_manager.cc:1918] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:28.615978 1812 ts_tablet_manager.cc:1918] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:28.616080 1812 log.cc:1196] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/eba1172c8f8348cc97494de9add2c4b0
I20260928 02:53:28.616254 1811 log.cc:1196] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/eba1172c8f8348cc97494de9add2c4b0
I20260928 02:53:28.616438 1812 ts_tablet_manager.cc:1939] T eba1172c8f8348cc97494de9add2c4b0 P 5960f06594b9411896209f57cafddc71: Deleting consensus metadata
I20260928 02:53:28.616719 1811 ts_tablet_manager.cc:1939] T eba1172c8f8348cc97494de9add2c4b0 P b25e2e97042b4a7e88236e2af97d41c8: Deleting consensus metadata
I20260928 02:53:28.617784 1297 catalog_manager.cc:4841] TS 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097): tablet eba1172c8f8348cc97494de9add2c4b0 (table test-table [id=cb94e2c44e324362b3d7a02a2da05713]) successfully deleted
I20260928 02:53:28.617846 1299 catalog_manager.cc:4841] TS b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109): tablet eba1172c8f8348cc97494de9add2c4b0 (table test-table [id=cb94e2c44e324362b3d7a02a2da05713]) successfully deleted
W20260928 02:53:28.619161 1298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:43699 (error 108)
I20260928 02:53:28.619323 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1508
I20260928 02:53:28.625015 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1642
I20260928 02:53:28.626034 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.625051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43699 (local address 127.0.0.1:37012)
0928 02:53:28.625802 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.625929 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:43699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:53:28.626240 1298 catalog_manager.cc:4584] TS b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eba1172c8f8348cc97494de9add2c4b0: Network error: Client connection negotiation failed: client connection to 127.19.139.129:43699: connect: Connection refused (error 111)
I20260928 02:53:28.631711 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1280
I20260928 02:53:28.637352 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41605
--webserver_interface=127.19.139.190
--webserver_port=41327
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:28.746976 1814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:28.747268 1814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:28.747308 1814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:28.751470 1814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:28.751545 1814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:28.751564 1814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:28.751579 1814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:28.751595 1814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:28.757259 1814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=41327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:28.758411 1814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:28.759505 1814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:28.772063 1823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:28.772064 1821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:28.772064 1820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:28.772149 1814 server_base.cc:1034] running on GCE node
I20260928 02:53:28.772986 1814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:28.774199 1814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:28.775378 1814 hybrid_clock.cc:648] HybridClock initialized: now 1790564008775358 us; error 33 us; skew 500 ppm
Sep 28 02:53:28 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564008, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:28.778347 1814 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:28.779346 1814 webserver.cc:460] Webserver started at http://127.19.139.190:41327/ using document root <none> and password file <none>
I20260928 02:53:28.779924 1814 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:28.780009 1814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:28.784129 1814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:28.786820 1830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:28.788488 1814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:28.788616 1814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "927572b566e847bfab871367885bbdef"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "726a0e6532743acd8abce3ba548cd3e9"
server_key_iv: "83af73dc518c10a09523003bd360a8d0"
server_key_version: "encryptionkey@0"
I20260928 02:53:28.789250 1814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:28.815011 1814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:28.818140 1814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:28.818343 1814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:28.825297 1814 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:41605
I20260928 02:53:28.825560 1882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:41605 every 8 connection(s)
I20260928 02:53:28.826447 1814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:28.830943 1883 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:53:28.832744 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap starting.
I20260928 02:53:28.833837 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1814
I20260928 02:53:28.834264 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:43699
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:41605
--webserver_port=34567
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:28.850421 1883 log.cc:824] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:28.860031 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:53:28.860611 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap complete.
I20260928 02:53:28.864816 1883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:28.865276 1883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927572b566e847bfab871367885bbdef, State: Initialized, Role: FOLLOWER
I20260928 02:53:28.865828 1883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:28.865967 1883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:28.866050 1883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:28.866160 1883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:53:28.868434 1883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:28.868824 1883 leader_election.cc:302] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 927572b566e847bfab871367885bbdef; no voters:
I20260928 02:53:28.869175 1883 leader_election.cc:288] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:53:28.869289 1887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:53:28.869535 1887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 LEADER]: Becoming Leader. State: Replica: 927572b566e847bfab871367885bbdef, State: Running, Role: LEADER
I20260928 02:53:28.869885 1887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:28.870585 1883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:28.871484 1889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927572b566e847bfab871367885bbdef. Latest consensus state: current_term: 2 leader_uuid: "927572b566e847bfab871367885bbdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } } }
I20260928 02:53:28.871523 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "927572b566e847bfab871367885bbdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } } }
I20260928 02:53:28.871620 1889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:28.871655 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:28.872433 1896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:28.877758 1896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:28.878623 1896 catalog_manager.cc:1259] Loaded cluster ID: e7b633dea144431995061677ff356006
I20260928 02:53:28.878703 1896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:28.879926 1896 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:53:28.882750 1896 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:53:28.887861 1896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:28.888736 1896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Loaded TSK: 0
I20260928 02:53:28.889755 1896 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:53:28.977072 1885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:28.977469 1885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:28.977535 1885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:28.983510 1885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:28.983613 1885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:28.983704 1885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:53:28.991593 1885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:43699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=34567
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:28.993013 1885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:28.994395 1885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:29.005699 1910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.005723 1911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.005998 1913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:29.005981 1885 server_base.cc:1034] running on GCE node
I20260928 02:53:29.006471 1885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:29.007155 1885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:29.008349 1885 hybrid_clock.cc:648] HybridClock initialized: now 1790564009008319 us; error 50 us; skew 500 ppm
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564009, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:29.011130 1885 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:53:29.012094 1885 webserver.cc:460] Webserver started at http://127.19.139.129:34567/ using document root <none> and password file <none>
I20260928 02:53:29.012702 1885 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:29.012778 1885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:29.016897 1885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:29.019162 1920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.020653 1885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:29.020758 1885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "b41748e7d1c747c891ba9579efae0377"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "06f5e3cfe879f28b6d33b0ef4aeeed5e"
server_key_iv: "e0e75fe9de8b1629619ffa215658bb1c"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.021291 1885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:29.036875 1885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:29.040141 1885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:29.040365 1885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:29.040999 1885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:29.042122 1927 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:53:29.047546 1885 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:53:29.047632 1885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 02:53:29.047700 1885 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:53:29.049575 1885 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:53:29.049628 1885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260928 02:53:29.049940 1927 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Bootstrap starting.
I20260928 02:53:29.059322 1885 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:43699
I20260928 02:53:29.059417 2034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:43699 every 8 connection(s)
I20260928 02:53:29.060427 1885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:29.061326 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 1885
I20260928 02:53:29.061734 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:35097
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:41605
--webserver_port=43939
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:29.069779 1927 log.cc:824] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564009, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:29.077860 1927 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:53:29.078294 1927 tablet_bootstrap.cc:492] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Bootstrap complete.
I20260928 02:53:29.079219 1927 ts_tablet_manager.cc:1397] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20260928 02:53:29.082144 1927 raft_consensus.cc:348] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:29.082734 1927 raft_consensus.cc:729] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41748e7d1c747c891ba9579efae0377, State: Initialized, Role: FOLLOWER
I20260928 02:53:29.083207 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.062818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:54549)
0928 02:53:29.065301 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.065365 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.066521 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.068129 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.068139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.068642 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:29.069303 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.069325 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.071497 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.071500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.072850 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.072861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.073839 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.073843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.074025 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.074770 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:29.074788 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:29.076813 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:29.079409 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.079415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.079429 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.079788 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.080286 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.080289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.080291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.080472 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.081212 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:29.081219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:29.081548 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.081752 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.082130 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"thread_start_us":154,"threads_started":1}
I20260928 02:53:29.083294 1927 consensus_queue.cc:260] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } } peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } }
I20260928 02:53:29.083768 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.062809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:54549 (local address 127.19.139.190:41605)
0928 02:53:29.064940 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.064966 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.065528 (+ 562us) server_negotiation.cc:407] Connection header received
0928 02:53:29.067089 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.067107 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.067591 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.068015 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.069507 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.071331 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.072971 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.073686 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.073851 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.077070 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.077077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.077095 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:29.077131 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.079254 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.079951 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.079957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.079963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.080146 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.080570 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.080574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.080575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.081077 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.081242 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.081927 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.082263 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":142,"threads_started":1}
I20260928 02:53:29.084471 1927 ts_tablet_manager.cc:1428] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260928 02:53:29.086215 2035 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41605
I20260928 02:53:29.086458 2035 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:29.086999 2035 heartbeater.cc:511] Master 127.19.139.190:41605 requested a full tablet report, sending...
I20260928 02:53:29.089430 1847 ts_manager.cc:194] Registered new tserver with Master: b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699)
I20260928 02:53:29.092751 1847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:54549
I20260928 02:53:29.095227 2035 heartbeater.cc:503] Master 127.19.139.190:41605 was elected leader, sending a full tablet report...
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564008, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:53:29.103114 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.092405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.19.139.129:43699)
0928 02:53:29.093235 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.093246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.093271 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:53:29.093348 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.093353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.093557 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.093690 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.094411 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.095569 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.096573 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.097032 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.097130 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.099448 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.099459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.099472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:29.099496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.101233 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.101729 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.101736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.101737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.101829 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.102082 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.102085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.102087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.102343 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.102459 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.102808 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.102974 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":99,"threads_started":1}
I20260928 02:53:29.103104 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.092315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43699 (local address 127.0.0.1:37026)
0928 02:53:29.092937 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.092982 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.093112 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.093693 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.093699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.094143 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:29.094283 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.094294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.095697 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.095701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.096471 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.096482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.097152 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.097156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.097262 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.097762 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:29.097773 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:29.099306 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:29.101362 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.101372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.101378 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.101614 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.101918 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.101921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.101924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.102003 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.102453 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:29.102459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:29.102581 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.102703 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.102919 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":114,"threads_started":1}
I20260928 02:53:29.104406 1969 tablet_service.cc:1514] Processing DeleteTablet for tablet eba1172c8f8348cc97494de9add2c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:28 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:37026
I20260928 02:53:29.105463 2044 tablet_replica.cc:330] stopping tablet replica
I20260928 02:53:29.105763 2044 raft_consensus.cc:2229] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:29.106046 2044 raft_consensus.cc:2258] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:29.107043 2044 ts_tablet_manager.cc:1905] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:29.110566 2044 ts_tablet_manager.cc:1918] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:29.110661 2044 log.cc:1196] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/eba1172c8f8348cc97494de9add2c4b0
I20260928 02:53:29.110888 2044 ts_tablet_manager.cc:1939] T eba1172c8f8348cc97494de9add2c4b0 P b41748e7d1c747c891ba9579efae0377: Deleting consensus metadata
I20260928 02:53:29.112919 1833 catalog_manager.cc:4841] TS b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699): tablet eba1172c8f8348cc97494de9add2c4b0 (table test-table [id=cb94e2c44e324362b3d7a02a2da05713]) successfully deleted
W20260928 02:53:29.182170 2038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:29.182462 2038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:29.182505 2038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:29.186751 2038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:29.186832 2038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:29.186894 2038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:53:29.192971 2038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:35097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=43939
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:29.194150 2038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:29.195238 2038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:29.205708 2049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.205713 2052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:29.205806 2038 server_base.cc:1034] running on GCE node
W20260928 02:53:29.205708 2050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:29.206398 2038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:29.207072 2038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:29.208218 2038 hybrid_clock.cc:648] HybridClock initialized: now 1790564009208183 us; error 48 us; skew 500 ppm
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564009, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:29.211100 2038 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:53:29.212038 2038 webserver.cc:460] Webserver started at http://127.19.139.130:43939/ using document root <none> and password file <none>
I20260928 02:53:29.212637 2038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:29.212724 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:29.216738 2038 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:29.218873 2059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.220285 2038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:29.220409 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "5960f06594b9411896209f57cafddc71"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "4243f9965cddef8188fc9f7de1c51f80"
server_key_iv: "92fdfe15bdb606e8ae207dda6a0edfd3"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.220965 2038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:29.237401 2038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:29.240463 2038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:29.240639 2038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:29.241225 2038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:29.242056 2038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:29.242102 2038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.242154 2038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:29.242182 2038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.250622 2038 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:35097
I20260928 02:53:29.250648 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:35097 every 8 connection(s)
I20260928 02:53:29.251590 2038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:29.259133 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2038
I20260928 02:53:29.259563 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:38109
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:41605
--webserver_port=43715
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564009, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:29.265084 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.253312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:55917 (local address 127.19.139.190:41605)
0928 02:53:29.253436 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.253440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.254260 (+ 820us) server_negotiation.cc:407] Connection header received
0928 02:53:29.255653 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.255657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.255715 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.255789 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.257067 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.257589 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.258702 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.258934 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.259025 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.261661 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.261664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.261666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:29.261693 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.263047 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.263544 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.263548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.263550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.263627 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.263958 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.263961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.263963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.264146 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.264266 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.264772 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.264918 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:53:29.265925 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.253668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:55917)
0928 02:53:29.254110 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.254152 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.255423 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.256011 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.256018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.256405 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:29.256913 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.256926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.257705 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.257707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.258580 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.258589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.259072 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.259076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.259257 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.259899 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:29.259915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:29.261489 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:29.263163 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.263169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.263184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.263452 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.263708 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.263711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.263713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.263870 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.264275 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:29.264283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:29.264538 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.264710 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.265052 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260928 02:53:29.267380 2173 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41605
I20260928 02:53:29.267628 2173 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:29.268213 2173 heartbeater.cc:511] Master 127.19.139.190:41605 requested a full tablet report, sending...
I20260928 02:53:29.269361 1847 ts_manager.cc:194] Registered new tserver with Master: 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
I20260928 02:53:29.269936 1847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:55917
W20260928 02:53:29.371538 2176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:29.371822 2176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:29.371881 2176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:29.376075 2176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:29.376147 2176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:29.376259 2176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:53:29.381871 2176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:38109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=43715
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:29.383066 2176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:29.384305 2176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:29.394462 2182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.394580 2181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.394584 2184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:29.394696 2176 server_base.cc:1034] running on GCE node
I20260928 02:53:29.395282 2176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:29.395982 2176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:29.397174 2176 hybrid_clock.cc:648] HybridClock initialized: now 1790564009397150 us; error 42 us; skew 500 ppm
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564009, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:29.399968 2176 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:53:29.400962 2176 webserver.cc:460] Webserver started at http://127.19.139.131:43715/ using document root <none> and password file <none>
I20260928 02:53:29.401533 2176 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:29.401623 2176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:29.405893 2176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:53:29.408365 2191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.409668 2176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:29.409790 2176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "b25e2e97042b4a7e88236e2af97d41c8"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "c30d02390d04379e5ace8b56321b0d2d"
server_key_iv: "a5ac1a80186d82e6a726f7ee0dc79de5"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.410362 2176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:29.426966 2176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:29.430174 2176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:29.430356 2176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:29.431010 2176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:29.431854 2176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:29.431900 2176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.431953 2176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:29.431978 2176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.440474 2176 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:38109
I20260928 02:53:29.440505 2304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:38109 every 8 connection(s)
I20260928 02:53:29.441479 2176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:29.448818 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2176
Sep 28 02:53:29 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564009, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:29.455551 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.443227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:38165 (local address 127.19.139.190:41605)
0928 02:53:29.443371 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.443375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.444187 (+ 812us) server_negotiation.cc:407] Connection header received
0928 02:53:29.445551 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.445554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.445604 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.445681 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.446833 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.447361 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.448521 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.448839 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.448950 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.451725 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.451728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.451730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:29.451754 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.453344 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.453878 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.453883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.453884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.453979 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.454306 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.454310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.454312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.454502 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.454577 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.455287 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.455414 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:53:29.456799 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.443572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:38165)
0928 02:53:29.444001 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.444045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.445320 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.445845 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.445852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.446200 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:29.446710 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.446724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.447452 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.447454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.448392 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.448402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.449014 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.449019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.449179 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.449842 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:29.449862 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:29.451561 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:29.453449 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.453456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.453475 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.453772 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.454073 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:29.454075 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:29.454077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:29.454219 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:29.454612 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:29.454620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:29.454996 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.455222 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.455584 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260928 02:53:29.457762 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.449609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.19.139.190:41605)
0928 02:53:29.449849 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.449851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.449964 (+ 113us) server_negotiation.cc:407] Connection header received
0928 02:53:29.450088 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.450090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.450127 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.450198 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.451011 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.451593 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.452608 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.452982 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.453094 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.454404 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.454407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.454409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:29.454432 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.456002 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.456506 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.456510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.456512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.456600 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:29.456832 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:29.456835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:29.456836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:29.457044 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.457246 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.457478 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.457605 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":55,"threads_started":1}
I20260928 02:53:29.458127 2305 heartbeater.cc:347] Connected to a master server at 127.19.139.190:41605
I20260928 02:53:29.458385 2305 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:29.458923 2305 heartbeater.cc:511] Master 127.19.139.190:41605 requested a full tablet report, sending...
I20260928 02:53:29.459991 1847 ts_manager.cc:194] Registered new tserver with Master: b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109)
I20260928 02:53:29.460611 1847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:38165
I20260928 02:53:29.471654 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:53:29.475970 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.472376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.19.139.190:41605)
0928 02:53:29.472497 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.472501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.472714 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:53:29.472859 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.472861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.472970 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.473061 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:29.473308 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.473838 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.474660 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.474903 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.474988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.475241 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:29.475657 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:29.475747 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.475793 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.475853 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:53:29.476683 1847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:29.479974 1847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:29.488121 1969 tablet_service.cc:1467] Processing CreateTablet for tablet 61794103ec7f42f2b37c3838b03d1eaa (DEFAULT_TABLE table=test-table [id=95f402c1a5204bd29695ecac50a76b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:29.488775 1969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61794103ec7f42f2b37c3838b03d1eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:29.491616 1927 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Bootstrap starting.
I20260928 02:53:29.492482 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.487008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:38109 (local address 127.0.0.1:35280)
0928 02:53:29.487164 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.487180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.487257 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.488048 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.488051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.488067 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:29.488206 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.488213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.489536 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.489540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.490733 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.490744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.492000 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.492004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.492112 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.492189 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.492261 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.492350 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:53:29.492619 1927 tablet_bootstrap.cc:654] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:29.492713 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.487201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.19.139.131:38109)
0928 02:53:29.487612 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.487625 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.487648 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:53:29.487721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.487725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.487906 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.488048 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:29.488315 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.489418 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.490865 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.491907 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.492021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.492228 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.492447 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.492560 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":167,"threads_started":1}
I20260928 02:53:29.493808 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.487311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35097 (local address 127.0.0.1:59936)
0928 02:53:29.487822 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.487836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.487924 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.489099 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.489102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.489121 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:29.489216 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.489222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.490641 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.490645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.491771 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.491782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.493387 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.493391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.493482 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.493504 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.493593 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.493673 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":80,"threads_started":1}
I20260928 02:53:29.494225 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.488106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.19.139.130:35097)
0928 02:53:29.488624 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.488635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.488674 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:53:29.488749 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.488754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.488951 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.489107 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:29.489327 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.490516 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.491909 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.493269 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.493389 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.493518 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.493975 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.494074 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":143,"threads_started":1}
I20260928 02:53:29.494524 1927 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: No bootstrap required, opened a new log
I20260928 02:53:29.494602 1927 ts_tablet_manager.cc:1397] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:29.495004 1927 raft_consensus.cc:348] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.495105 1927 raft_consensus.cc:374] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:29.495132 1927 raft_consensus.cc:729] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41748e7d1c747c891ba9579efae0377, State: Initialized, Role: FOLLOWER
I20260928 02:53:29.495266 1927 consensus_queue.cc:260] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.495537 1927 ts_tablet_manager.cc:1428] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:53:29.496003 2239 tablet_service.cc:1467] Processing CreateTablet for tablet 61794103ec7f42f2b37c3838b03d1eaa (DEFAULT_TABLE table=test-table [id=95f402c1a5204bd29695ecac50a76b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:29.497226 2239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61794103ec7f42f2b37c3838b03d1eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:29.498435 2107 tablet_service.cc:1467] Processing CreateTablet for tablet 61794103ec7f42f2b37c3838b03d1eaa (DEFAULT_TABLE table=test-table [id=95f402c1a5204bd29695ecac50a76b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:29.499368 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61794103ec7f42f2b37c3838b03d1eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:29.504567 2318 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Bootstrap starting.
I20260928 02:53:29.505883 2319 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Bootstrap starting.
I20260928 02:53:29.507812 2318 tablet_bootstrap.cc:654] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:29.508543 2319 tablet_bootstrap.cc:654] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:29.508798 2318 log.cc:824] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:29.509377 2319 log.cc:824] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:29.511004 2318 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: No bootstrap required, opened a new log
I20260928 02:53:29.511231 2318 ts_tablet_manager.cc:1397] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:53:29.511469 2319 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: No bootstrap required, opened a new log
I20260928 02:53:29.511647 2319 ts_tablet_manager.cc:1397] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:53:29.515240 2318 raft_consensus.cc:348] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.515388 2319 raft_consensus.cc:348] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.515540 2318 raft_consensus.cc:374] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:29.515586 2319 raft_consensus.cc:374] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:29.515609 2318 raft_consensus.cc:729] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25e2e97042b4a7e88236e2af97d41c8, State: Initialized, Role: FOLLOWER
I20260928 02:53:29.515637 2319 raft_consensus.cc:729] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5960f06594b9411896209f57cafddc71, State: Initialized, Role: FOLLOWER
I20260928 02:53:29.516091 2319 consensus_queue.cc:260] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.516091 2318 consensus_queue.cc:260] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.516837 2305 heartbeater.cc:503] Master 127.19.139.190:41605 was elected leader, sending a full tablet report...
I20260928 02:53:29.517082 2173 heartbeater.cc:503] Master 127.19.139.190:41605 was elected leader, sending a full tablet report...
I20260928 02:53:29.517189 2318 ts_tablet_manager.cc:1428] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:53:29.518983 2319 ts_tablet_manager.cc:1428] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260928 02:53:29.562817 2036 tablet.cc:2369] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:53:29.692483 2306 tablet.cc:2369] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:53:29.752792 2174 tablet.cc:2369] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:29.828560 2041 raft_consensus.cc:482] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:29.828843 2041 raft_consensus.cc:504] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.830058 2041 leader_election.cc:288] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109), 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
I20260928 02:53:29.834194 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.830231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:38109 (local address 127.19.139.129:38041)
0928 02:53:29.830576 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.830591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.830770 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.831138 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.831142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.831160 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:29.831283 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.831288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.832264 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.832268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.833059 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.833068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.833735 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.833739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.833843 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.833901 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.833973 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.834045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":85,"threads_started":1}
I20260928 02:53:29.834193 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.830472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:38041 (local address 127.19.139.131:38109)
0928 02:53:29.830587 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.830591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.830677 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:53:29.830931 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.830934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.830973 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.831034 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:29.831383 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.832139 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.833176 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.833615 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.833740 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.833801 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.833994 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.834070 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:53:29.834815 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.830306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35097 (local address 127.19.139.129:53267)
0928 02:53:29.830711 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.830723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.830800 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.831040 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.831043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.831059 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:29.831157 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.831162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.832174 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.832184 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.833398 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.833408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.834398 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.834402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.834493 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.834517 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.834600 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.834678 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":60,"threads_started":1}
I20260928 02:53:29.834826 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.830610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:53267 (local address 127.19.139.130:35097)
0928 02:53:29.830719 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.830723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.830790 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:53:29.830890 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.830894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.830941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.831018 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:29.831280 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.832044 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.833520 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.834041 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.834171 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.834216 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.834637 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.834717 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:53:29.834985 2259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61794103ec7f42f2b37c3838b03d1eaa" candidate_uuid: "b41748e7d1c747c891ba9579efae0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25e2e97042b4a7e88236e2af97d41c8" is_pre_election: true
I20260928 02:53:29.835342 2259 raft_consensus.cc:2454] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41748e7d1c747c891ba9579efae0377 in term 0.
I20260928 02:53:29.835656 2127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61794103ec7f42f2b37c3838b03d1eaa" candidate_uuid: "b41748e7d1c747c891ba9579efae0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5960f06594b9411896209f57cafddc71" is_pre_election: true
I20260928 02:53:29.835872 1924 leader_election.cc:302] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25e2e97042b4a7e88236e2af97d41c8, b41748e7d1c747c891ba9579efae0377; no voters:
I20260928 02:53:29.835953 2127 raft_consensus.cc:2454] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41748e7d1c747c891ba9579efae0377 in term 0.
I20260928 02:53:29.836184 2041 raft_consensus.cc:2781] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:29.836273 2041 raft_consensus.cc:482] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:29.836313 2041 raft_consensus.cc:3037] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:29.837551 2041 raft_consensus.cc:504] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.837910 2041 leader_election.cc:288] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 1 election: Requested vote from peers b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109), 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
I20260928 02:53:29.838310 2259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61794103ec7f42f2b37c3838b03d1eaa" candidate_uuid: "b41748e7d1c747c891ba9579efae0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25e2e97042b4a7e88236e2af97d41c8"
I20260928 02:53:29.838335 2127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61794103ec7f42f2b37c3838b03d1eaa" candidate_uuid: "b41748e7d1c747c891ba9579efae0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5960f06594b9411896209f57cafddc71"
I20260928 02:53:29.838439 2259 raft_consensus.cc:3037] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:29.838449 2127 raft_consensus.cc:3037] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:29.839434 2127 raft_consensus.cc:2454] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41748e7d1c747c891ba9579efae0377 in term 1.
I20260928 02:53:29.839476 2259 raft_consensus.cc:2454] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41748e7d1c747c891ba9579efae0377 in term 1.
I20260928 02:53:29.839702 1922 leader_election.cc:302] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5960f06594b9411896209f57cafddc71, b41748e7d1c747c891ba9579efae0377; no voters:
I20260928 02:53:29.839862 2041 raft_consensus.cc:2781] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:29.840049 2041 raft_consensus.cc:686] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 LEADER]: Becoming Leader. State: Replica: b41748e7d1c747c891ba9579efae0377, State: Running, Role: LEADER
I20260928 02:53:29.840334 2041 consensus_queue.cc:237] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:29.843339 1846 catalog_manager.cc:5495] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 reported cstate change: term changed from 0 to 1, leader changed from <none> to b41748e7d1c747c891ba9579efae0377 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "b41748e7d1c747c891ba9579efae0377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } health_report { overall_health: HEALTHY } } }
I20260928 02:53:29.857903 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.854259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.19.139.129:43699)
0928 02:53:29.854561 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.854569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.854580 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:53:29.854671 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.854674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.854726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.854857 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:29.855093 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.855611 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.856466 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.856691 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.856779 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.857041 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:29.857481 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:29.857589 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.857684 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.857743 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":116,"threads_started":1}
I20260928 02:53:29.861974 2127 raft_consensus.cc:1261] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Refusing update from remote peer b41748e7d1c747c891ba9579efae0377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:29.861974 2259 raft_consensus.cc:1261] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Refusing update from remote peer b41748e7d1c747c891ba9579efae0377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:29.862749 2326 consensus_queue.cc:1035] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:29.862987 2041 consensus_queue.cc:1035] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:29.879313 1846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:29.879483 1846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:29.882040 1846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Sending DeleteTablet for 3 replicas of tablet 61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:29.882786 2239 tablet_service.cc:1514] Processing DeleteTablet for tablet 61794103ec7f42f2b37c3838b03d1eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:29 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:35280
I20260928 02:53:29.882926 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1885
I20260928 02:53:29.883550 2107 tablet_service.cc:1514] Processing DeleteTablet for tablet 61794103ec7f42f2b37c3838b03d1eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:29 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:59936
I20260928 02:53:29.883761 2340 tablet_replica.cc:330] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: stopping tablet replica
I20260928 02:53:29.884073 2341 tablet_replica.cc:330] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: stopping tablet replica
I20260928 02:53:29.884109 2340 raft_consensus.cc:2229] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:29.884418 2341 raft_consensus.cc:2229] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:29.884466 2340 raft_consensus.cc:2258] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:29.884723 2341 raft_consensus.cc:2258] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:29.886256 2341 ts_tablet_manager.cc:1905] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:29.888639 2341 ts_tablet_manager.cc:1918] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:29.888742 2341 log.cc:1196] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:29.889056 2341 ts_tablet_manager.cc:1939] T 61794103ec7f42f2b37c3838b03d1eaa P 5960f06594b9411896209f57cafddc71: Deleting consensus metadata
W20260928 02:53:29.889254 1833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:43699 (error 108)
I20260928 02:53:29.889469 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 2038
I20260928 02:53:29.890414 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.890095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43699 (local address 127.0.0.1:37048)
0928 02:53:29.890240 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.890315 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:43699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:53:29.890589 1833 catalog_manager.cc:4584] TS b41748e7d1c747c891ba9579efae0377 (127.19.139.129:43699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61794103ec7f42f2b37c3838b03d1eaa: Network error: Client connection negotiation failed: client connection to 127.19.139.129:43699: connect: Connection refused (error 111)
I20260928 02:53:29.892009 2340 ts_tablet_manager.cc:1905] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:29.894380 2340 ts_tablet_manager.cc:1918] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:29.894433 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 2176
I20260928 02:53:29.894493 2340 log.cc:1196] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:29.895422 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.894598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35097 (local address 127.0.0.1:59942)
0928 02:53:29.895285 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.895341 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607}
I20260928 02:53:29.899525 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 1814
I20260928 02:53:29.904632 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41605
--webserver_interface=127.19.139.190
--webserver_port=41327
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:30.013857 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:30.014146 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:30.014187 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:30.018457 2343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:30.018543 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:30.018563 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:30.018579 2343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:30.018592 2343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:30.024252 2343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=41327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:30.025391 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:30.026481 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:30.035120 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.035203 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.035120 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.035356 2343 server_base.cc:1034] running on GCE node
I20260928 02:53:30.036041 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:30.037171 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:30.038332 2343 hybrid_clock.cc:648] HybridClock initialized: now 1790564010038316 us; error 30 us; skew 500 ppm
Sep 28 02:53:30 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564010, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:30.041097 2343 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:30.042044 2343 webserver.cc:460] Webserver started at http://127.19.139.190:41327/ using document root <none> and password file <none>
I20260928 02:53:30.042588 2343 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:30.042657 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:30.046950 2343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:30.049288 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.050709 2343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:30.050843 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "927572b566e847bfab871367885bbdef"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "726a0e6532743acd8abce3ba548cd3e9"
server_key_iv: "83af73dc518c10a09523003bd360a8d0"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.051388 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:30.071209 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:30.074810 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:30.075028 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:30.082446 2343 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:41605
I20260928 02:53:30.082707 2410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:41605 every 8 connection(s)
I20260928 02:53:30.083634 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:30.088260 2411 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:53:30.090019 2411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap starting.
I20260928 02:53:30.091004 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2343
I20260928 02:53:30.091530 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:43699
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:41605
--webserver_port=34567
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:30.108064 2411 log.cc:824] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:30.119531 2411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:53:30.119935 2411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef: Bootstrap complete.
I20260928 02:53:30.123833 2411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:30.124312 2411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927572b566e847bfab871367885bbdef, State: Initialized, Role: FOLLOWER
I20260928 02:53:30.124797 2411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:30.124933 2411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:30.124998 2411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:30.125093 2411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:53:30.127095 2411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:30.127424 2411 leader_election.cc:302] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 927572b566e847bfab871367885bbdef; no voters:
I20260928 02:53:30.127722 2411 leader_election.cc:288] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:53:30.127823 2415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:53:30.128049 2415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [term 3 LEADER]: Becoming Leader. State: Replica: 927572b566e847bfab871367885bbdef, State: Running, Role: LEADER
I20260928 02:53:30.128397 2415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } }
I20260928 02:53:30.129022 2411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:30.129694 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "927572b566e847bfab871367885bbdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } } }
I20260928 02:53:30.129823 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:30.130137 2417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927572b566e847bfab871367885bbdef. Latest consensus state: current_term: 3 leader_uuid: "927572b566e847bfab871367885bbdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927572b566e847bfab871367885bbdef" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 41605 } } }
I20260928 02:53:30.130254 2417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927572b566e847bfab871367885bbdef [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:30.130801 2422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:30.136026 2422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:30.136767 2422 catalog_manager.cc:1259] Loaded cluster ID: e7b633dea144431995061677ff356006
I20260928 02:53:30.136837 2422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:53:30.137866 2422 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f62544169ad google::LogMessage::Fail() at ??:0
@ 0x7f625441988a google::LogMessage::SendToLog() at ??:0
@ 0x7f62544164de google::LogMessage::Flush() at ??:0
@ 0x7f6254417a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f625ed55fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f625ed56d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f625ed53579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f625ed97640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f62566683d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6256668c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f625666a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6256659512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6257c9d6db start_thread at ??:0
@ 0x7f6252f1461f clone at ??:0
W20260928 02:53:30.226822 2413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:30.227102 2413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:30.227155 2413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:30.231320 2413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:30.231398 2413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:30.231460 2413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:53:30.236981 2413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:43699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=34567
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:30.238118 2413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:30.239176 2413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:30.249049 2453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.249011 2452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.249054 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.249578 2413 server_base.cc:1034] running on GCE node
I20260928 02:53:30.250182 2413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:30.251061 2413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:30.252243 2413 hybrid_clock.cc:648] HybridClock initialized: now 1790564010252204 us; error 53 us; skew 500 ppm
Sep 28 02:53:30 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564010, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:30.255787 2413 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:53:30.257135 2413 webserver.cc:460] Webserver started at http://127.19.139.129:34567/ using document root <none> and password file <none>
I20260928 02:53:30.257921 2413 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:30.258009 2413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:30.263199 2413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 02:53:30.266045 2462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.267586 2413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:30.267733 2413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "b41748e7d1c747c891ba9579efae0377"
format_stamp: "Formatted at 2026-09-28 02:53:27 on dist-test-slave-5vc0"
server_key: "06f5e3cfe879f28b6d33b0ef4aeeed5e"
server_key_iv: "e0e75fe9de8b1629619ffa215658bb1c"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.268502 2413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:30.303246 2413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:30.307327 2413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:30.307526 2413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:30.308290 2413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:30.309641 2469 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:53:30.313889 2413 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:53:30.313977 2413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 02:53:30.314049 2413 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:53:30.316529 2413 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:53:30.316596 2413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:30.316937 2469 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Bootstrap starting.
I20260928 02:53:30.327630 2413 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:43699
I20260928 02:53:30.328008 2576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:43699 every 8 connection(s)
I20260928 02:53:30.328949 2413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:30.336246 2469 log.cc:824] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:30.339143 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2413
I20260928 02:53:30.339670 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:35097
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:41605
--webserver_port=43939
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:30.344019 2469 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:53:30.344393 2469 tablet_bootstrap.cc:492] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Bootstrap complete.
I20260928 02:53:30.345305 2469 ts_tablet_manager.cc:1397] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.015s
I20260928 02:53:30.347779 2469 raft_consensus.cc:348] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:30.348222 2469 raft_consensus.cc:729] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41748e7d1c747c891ba9579efae0377, State: Initialized, Role: FOLLOWER
I20260928 02:53:30.348687 2469 consensus_queue.cc:260] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:30.349601 2469 ts_tablet_manager.cc:1428] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
W20260928 02:53:30.382602 2579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:30.331367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:54475)
0928 02:53:30.332091 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.332205 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:30.333354 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:30.381641 (+ 48287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":157,"threads_started":1}
W20260928 02:53:30.383155 2577 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:41605 (0 consecutive failures): Network error: Failed to ping master at 127.19.139.190:41605: Client connection negotiation failed: client connection to 127.19.139.190:41605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:53:30.383941 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.383647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:32791)
0928 02:53:30.383793 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.383870 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:53:30.384683 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.384455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:56531)
0928 02:53:30.384569 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.384620 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:53:30.384858 2577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:53:30.458485 2581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:30.458760 2581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:30.458802 2581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:30.462777 2581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:30.462838 2581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:30.462898 2581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:53:30.468122 2581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:35097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=43939
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:30.469198 2581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:30.470181 2581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:30.480129 2587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.480211 2590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.480252 2581 server_base.cc:1034] running on GCE node
W20260928 02:53:30.480140 2588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.480789 2581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:30.481438 2581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:30.482622 2581 hybrid_clock.cc:648] HybridClock initialized: now 1790564010482588 us; error 44 us; skew 500 ppm
Sep 28 02:53:30 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564010, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:30.485517 2581 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:53:30.486447 2581 webserver.cc:460] Webserver started at http://127.19.139.130:43939/ using document root <none> and password file <none>
I20260928 02:53:30.487044 2581 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:30.487109 2581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:30.491122 2581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:53:30.493296 2597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.494645 2581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:53:30.494768 2581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "5960f06594b9411896209f57cafddc71"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "4243f9965cddef8188fc9f7de1c51f80"
server_key_iv: "92fdfe15bdb606e8ae207dda6a0edfd3"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.495321 2581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:30.510696 2581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:30.513516 2581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:30.513693 2581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:30.514267 2581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:30.515065 2581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:30.515111 2581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.515149 2581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:30.515166 2581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.523941 2581 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:35097
I20260928 02:53:30.523979 2710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:35097 every 8 connection(s)
I20260928 02:53:30.525027 2581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:30.529045 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2581
I20260928 02:53:30.529270 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.527229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:41887)
0928 02:53:30.527711 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.527877 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260928 02:53:30.529560 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:38109
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:41605
--webserver_port=43715
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:30.529724 2711 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:41605 (0 consecutive failures): Network error: Failed to ping master at 127.19.139.190:41605: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
I20260928 02:53:30.530373 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.530132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:52453)
0928 02:53:30.530248 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.530306 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:53:30.531003 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.530799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:57361)
0928 02:53:30.530908 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.530952 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 02:53:30.531152 2711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:53:30.639896 2714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:30.640169 2714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:30.640234 2714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:30.644309 2714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:30.644397 2714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:30.644483 2714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:53:30.649813 2714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:38109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=43715
--tserver_master_addrs=127.19.139.190:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:30.650887 2714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:30.651976 2714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:30.661759 2722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.661763 2720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.661759 2719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.662019 2714 server_base.cc:1034] running on GCE node
I20260928 02:53:30.662549 2714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:30.663187 2714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:30.664359 2714 hybrid_clock.cc:648] HybridClock initialized: now 1790564010664334 us; error 39 us; skew 500 ppm
Sep 28 02:53:30 dist-test-slave-5vc0 krb5kdc[1267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564010, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:30.667296 2714 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:53:30.668306 2714 webserver.cc:460] Webserver started at http://127.19.139.131:43715/ using document root <none> and password file <none>
I20260928 02:53:30.668872 2714 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:30.668946 2714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:30.673012 2714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:53:30.675261 2729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.676658 2714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:30.676766 2714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "b25e2e97042b4a7e88236e2af97d41c8"
format_stamp: "Formatted at 2026-09-28 02:53:28 on dist-test-slave-5vc0"
server_key: "c30d02390d04379e5ace8b56321b0d2d"
server_key_iv: "a5ac1a80186d82e6a726f7ee0dc79de5"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.677302 2714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:30.690387 2714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:30.693874 2714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:30.694056 2714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:30.694640 2714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:30.695808 2736 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 02:53:30.699975 2736 ts_tablet_manager.cc:1852] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 02:53:30.700484 2736 ts_tablet_manager.cc:1905] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:30.704768 2736 ts_tablet_manager.cc:1918] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:30.704981 2736 ts_tablet_manager.cc:1939] T 61794103ec7f42f2b37c3838b03d1eaa P b25e2e97042b4a7e88236e2af97d41c8: Deleting consensus metadata
I20260928 02:53:30.705404 2714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 02:53:30.705478 2714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260928 02:53:30.705542 2714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:30.705575 2714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.714116 2714 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:38109
I20260928 02:53:30.714193 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:38109 every 8 connection(s)
I20260928 02:53:30.715011 2714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:30.716053 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2714
I20260928 02:53:30.718757 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.717451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:35959)
0928 02:53:30.717929 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.718037 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
W20260928 02:53:30.719165 2844 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:41605 (0 consecutive failures): Network error: Failed to ping master at 127.19.139.190:41605: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
I20260928 02:53:30.719801 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.719543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:59939)
0928 02:53:30.719695 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.719744 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:53:30.720441 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.720266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:52551)
0928 02:53:30.720351 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.720392 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:53:30.720593 2844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:53:31.385902 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.385451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:36785)
0928 02:53:31.385724 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:31.385791 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260928 02:53:31.532055 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.531666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:35979)
0928 02:53:31.531891 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:31.531951 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260928 02:53:31.721643 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.721200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:39589)
0928 02:53:31.721471 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:31.721535 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260928 02:53:31.896529 2851 raft_consensus.cc:482] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:31.896726 2851 raft_consensus.cc:504] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:31.897850 2851 leader_election.cc:288] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109), 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
Sep 28 02:53:31 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564010, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.131@KRBTEST.COM
Sep 28 02:53:31 dist-test-slave-5vc0 krb5kdc[1267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564010, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.130@KRBTEST.COM
I20260928 02:53:31.911244 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.898106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:35097 (local address 127.19.139.129:55699)
0928 02:53:31.898476 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:31.898490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:31.898590 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:31.900191 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:31.900197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:31.900716 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:31.901196 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:31.901213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.902466 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.902469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:31.903408 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:31.903418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.904100 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.904104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:31.904354 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:31.904972 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:31.905011 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:31.907043 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:31.909092 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:31.909096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:31.909320 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:31.909648 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:31.909651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:31.909737 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:31.910300 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:31.910305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:31.910534 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 02:53:31.910696 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 02:53:31.911053 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":185,"spinlock_wait_cycles":71424,"thread_start_us":73,"threads_started":1}
I20260928 02:53:31.911247 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.898248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:56159 (local address 127.19.139.131:38109)
0928 02:53:31.898668 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:31.898680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:31.898716 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:53:31.899461 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:31.899469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:31.899811 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:31.900081 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:31.901384 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.902331 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.903524 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.903984 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.904128 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:31.906925 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:31.906932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:31.906949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:31.906986 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:31.908627 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909241 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:31.909247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:31.909253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:31.909429 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909766 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:31.909769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:31.909771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:31.910188 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:31.910306 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:31.910821 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 02:53:31.911077 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":75,"threads_started":1}
I20260928 02:53:31.911263 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.898274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:55699 (local address 127.19.139.130:35097)
0928 02:53:31.898677 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:31.898695 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:31.898727 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:53:31.899395 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:31.899403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:31.899806 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:31.900081 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:31.901368 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.902332 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.903529 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.903986 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.904135 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:31.907225 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:31.907230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:31.907247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:31.907281 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:31.908973 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909418 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:31.909422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:31.909426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:31.909558 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909829 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:31.909832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:31.909833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:31.910187 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:31.910308 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:31.910820 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 02:53:31.911072 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":70,"threads_started":1}
I20260928 02:53:31.911276 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.898055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:38109 (local address 127.19.139.129:56159)
0928 02:53:31.898400 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:31.898414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:31.898516 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:31.900191 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:31.900197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:31.900715 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:31.901196 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:31.901212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.902464 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.902467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:31.903413 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:31.903423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:31.904100 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:31.904103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:31.904352 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:31.905008 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:31.905019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:31.906665 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:31.908755 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:31.908770 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:31.908783 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:31.909136 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:31.909512 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:31.909515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:31.909517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:31.909681 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:31.910299 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:31.910305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:31.910534 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 02:53:31.910695 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:53:31.911127 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":300,"spinlock_wait_cycles":34304,"thread_start_us":75,"threads_started":1}
W20260928 02:53:31.913796 2466 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
W20260928 02:53:31.914016 2464 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:31.914063 2464 leader_election.cc:302] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b41748e7d1c747c891ba9579efae0377; no voters: 5960f06594b9411896209f57cafddc71, b25e2e97042b4a7e88236e2af97d41c8
I20260928 02:53:31.914306 2851 raft_consensus.cc:2726] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:53:32.387123 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.386802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:49835)
0928 02:53:32.386964 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:32.387034 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:53:32.533267 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.532780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:48045)
0928 02:53:32.533074 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:32.533133 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260928 02:53:32.723111 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.722596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:47147)
0928 02:53:32.722941 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:32.723011 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":147,"threads_started":1}
I20260928 02:53:33.388640 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.388072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:55199)
0928 02:53:33.388421 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:33.388502 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":116,"threads_started":1}
I20260928 02:53:33.534703 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.534137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:52411)
0928 02:53:33.534463 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:33.534599 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260928 02:53:33.724419 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.723977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:45577)
0928 02:53:33.724245 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:33.724320 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260928 02:53:33.931536 2861 raft_consensus.cc:482] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:33.931677 2861 raft_consensus.cc:504] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:33.932098 2861 leader_election.cc:288] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109), 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
W20260928 02:53:33.932853 2464 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
W20260928 02:53:33.932960 2466 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:33.933064 2466 leader_election.cc:302] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b41748e7d1c747c891ba9579efae0377; no voters: 5960f06594b9411896209f57cafddc71, b25e2e97042b4a7e88236e2af97d41c8
I20260928 02:53:33.933208 2861 raft_consensus.cc:2726] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:53:34.389935 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:34.389458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:39603)
0928 02:53:34.389743 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:34.389834 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260928 02:53:34.536207 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:34.535630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:48861)
0928 02:53:34.535979 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:34.536059 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":136,"threads_started":1}
I20260928 02:53:34.725521 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:34.725104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:48837)
0928 02:53:34.725357 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:34.725421 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260928 02:53:35.391322 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:35.390851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:34203)
0928 02:53:35.391163 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:35.391225 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":78,"threads_started":1}
I20260928 02:53:35.537562 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:35.537091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:45709)
0928 02:53:35.537385 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:35.537455 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260928 02:53:35.726718 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:35.726251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:37153)
0928 02:53:35.726556 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:35.726620 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260928 02:53:36.392602 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.392138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:50579)
0928 02:53:36.392442 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:36.392505 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260928 02:53:36.538869 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.538402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:59555)
0928 02:53:36.538693 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:36.538761 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260928 02:53:36.728072 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.727598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:49055)
0928 02:53:36.727901 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:36.727971 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
I20260928 02:53:37.014570 2871 raft_consensus.cc:482] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:37.014714 2871 raft_consensus.cc:504] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:37.015185 2871 leader_election.cc:288] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109), 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
W20260928 02:53:37.015992 2464 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
W20260928 02:53:37.016121 2466 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:37.016232 2466 leader_election.cc:302] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b41748e7d1c747c891ba9579efae0377; no voters: 5960f06594b9411896209f57cafddc71, b25e2e97042b4a7e88236e2af97d41c8
I20260928 02:53:37.016359 2871 raft_consensus.cc:2726] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:53:37.393770 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.393363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:34891)
0928 02:53:37.393609 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.393671 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260928 02:53:37.540345 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.539764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:60421)
0928 02:53:37.540128 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.540232 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
I20260928 02:53:37.729494 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.728985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:45319)
0928 02:53:37.729327 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.729396 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":78,"threads_started":1}
I20260928 02:53:38.394881 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:38.394499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:56157)
0928 02:53:38.394723 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:38.394777 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260928 02:53:38.541599 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:38.541165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:37119)
0928 02:53:38.541433 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:38.541499 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260928 02:53:38.730894 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:38.730438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:34105)
0928 02:53:38.730714 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:38.730783 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260928 02:53:39.396232 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:39.395687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:48187)
0928 02:53:39.396018 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:39.396082 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":89,"threads_started":1}
I20260928 02:53:39.542791 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:39.542381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:53731)
0928 02:53:39.542623 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:39.542686 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260928 02:53:39.732291 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:39.731810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:55307)
0928 02:53:39.732080 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:39.732146 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260928 02:53:40.397601 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:40.397122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:38881)
0928 02:53:40.397427 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:40.397493 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260928 02:53:40.544298 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:40.543797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:57913)
0928 02:53:40.544097 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:40.544195 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260928 02:53:40.733597 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:40.733140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:53719)
0928 02:53:40.733432 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:40.733500 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
I20260928 02:53:41.399061 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:41.398570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:42451)
0928 02:53:41.398895 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:41.398961 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
I20260928 02:53:41.519286 2885 raft_consensus.cc:482] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:41.519434 2885 raft_consensus.cc:504] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25e2e97042b4a7e88236e2af97d41c8" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 38109 } } peers { permanent_uuid: "5960f06594b9411896209f57cafddc71" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 35097 } } peers { permanent_uuid: "b41748e7d1c747c891ba9579efae0377" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43699 } }
I20260928 02:53:41.519917 2885 leader_election.cc:288] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109), 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097)
W20260928 02:53:41.520767 2464 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5960f06594b9411896209f57cafddc71 (127.19.139.130:35097): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
W20260928 02:53:41.520877 2466 leader_election.cc:341] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b25e2e97042b4a7e88236e2af97d41c8 (127.19.139.131:38109): Not found: Tablet not found: 61794103ec7f42f2b37c3838b03d1eaa
I20260928 02:53:41.520978 2466 leader_election.cc:302] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b41748e7d1c747c891ba9579efae0377; no voters: 5960f06594b9411896209f57cafddc71, b25e2e97042b4a7e88236e2af97d41c8
I20260928 02:53:41.521118 2885 raft_consensus.cc:2726] T 61794103ec7f42f2b37c3838b03d1eaa P b41748e7d1c747c891ba9579efae0377 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:53:41.545680 2886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:41.545201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:37157)
0928 02:53:41.545508 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:41.545575 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
I20260928 02:53:41.735033 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:41.734529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:46159)
0928 02:53:41.734861 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:41.734935 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260928 02:53:42.400429 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:42.399894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:40427)
0928 02:53:42.400230 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:42.400314 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
I20260928 02:53:42.547066 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:42.546548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:40507)
0928 02:53:42.546899 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:42.546967 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260928 02:53:42.736531 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:42.735967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:41379)
0928 02:53:42.736346 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:42.736429 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
I20260928 02:53:43.401628 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:43.401198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:47357)
0928 02:53:43.401477 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:43.401528 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260928 02:53:43.548597 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:43.548074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:42123)
0928 02:53:43.548433 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:43.548499 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260928 02:53:43.737768 2893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:43.737292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:59765)
0928 02:53:43.737609 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:43.737670 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":76,"threads_started":1}
I20260928 02:53:44.402978 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:44.402492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:33069)
0928 02:53:44.402806 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:44.402877 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260928 02:53:44.549769 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:44.549330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:34995)
0928 02:53:44.549604 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:44.549666 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260928 02:53:44.738970 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:44.738540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.131:43613)
0928 02:53:44.738804 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:44.738875 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260928 02:53:45.404523 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:45.404009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.129:37477)
0928 02:53:45.404339 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:45.404420 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":79,"threads_started":1}
I20260928 02:53:45.551192 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:45.550739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:41605 (local address 127.19.139.130:50631)
0928 02:53:45.551027 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:45.551093 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
I20260928 02:53:45.726714 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 2413
I20260928 02:53:45.732102 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 2581
I20260928 02:53:45.736620 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 2714
2026-09-28T02:53:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19296 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:45 dist-test-slave-5vc0 krb5kdc[2901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:45 dist-test-slave-5vc0 krb5kdc[2901](info): set up 2 sockets
Sep 28 02:53:45 dist-test-slave-5vc0 krb5kdc[2901](info): commencing operation
krb5kdc: starting...
W20260928 02:53:46.700276 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:53:46 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:53:46.823240 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37411
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:46.935359 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:46.935644 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:46.935685 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:46.939793 2914 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:53:46.939857 2914 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:53:46.939872 2914 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:53:46.939885 2914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:46.939905 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:46.939936 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:46.939965 2914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:46.939989 2914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:46.945328 2914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:46.946373 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:46.947404 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:46.956521 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:46.956617 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:46.956521 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:46.956597 2914 server_base.cc:1034] running on GCE node
I20260928 02:53:46.957435 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:46.958603 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:46.959758 2914 hybrid_clock.cc:648] HybridClock initialized: now 1790564026959743 us; error 32 us; skew 500 ppm
Sep 28 02:53:46 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564026, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:46.962565 2914 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:53:46.963487 2914 webserver.cc:460] Webserver started at http://127.19.139.190:34651/ using document root <none> and password file <none>
I20260928 02:53:46.964037 2914 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:46.964098 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:46.964341 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:46.966532 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "cda06403ce1e41bfa66dc70a25c26d05"
format_stamp: "Formatted at 2026-09-28 02:53:46 on dist-test-slave-5vc0"
server_key: "8171132a6791ae481bafbd0629aa3413"
server_key_iv: "0fa154c74e7d1137b38671af0386122f"
server_key_version: "encryptionkey@0"
I20260928 02:53:46.967114 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "cda06403ce1e41bfa66dc70a25c26d05"
format_stamp: "Formatted at 2026-09-28 02:53:46 on dist-test-slave-5vc0"
server_key: "8171132a6791ae481bafbd0629aa3413"
server_key_iv: "0fa154c74e7d1137b38671af0386122f"
server_key_version: "encryptionkey@0"
I20260928 02:53:46.971081 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:53:46.973547 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:46.974951 2914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:53:46.975082 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "cda06403ce1e41bfa66dc70a25c26d05"
format_stamp: "Formatted at 2026-09-28 02:53:46 on dist-test-slave-5vc0"
server_key: "8171132a6791ae481bafbd0629aa3413"
server_key_iv: "0fa154c74e7d1137b38671af0386122f"
server_key_version: "encryptionkey@0"
I20260928 02:53:46.975219 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:46.994727 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:46.998070 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:46.998281 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:47.005203 2914 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:37411
I20260928 02:53:47.005461 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:37411 every 8 connection(s)
I20260928 02:53:47.006151 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:47.008997 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:47.009727 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 2914
I20260928 02:53:47.009837 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:53:47.010114 20014 external_mini_cluster.cc:1428] Setting key ab5b39004dbb84623185972c03801e39
I20260928 02:53:47.015486 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap starting.
I20260928 02:53:47.018091 2982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:47.018831 2982 log.cc:824] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:47.021035 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: No bootstrap required, opened a new log
I20260928 02:53:47.024989 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.011463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.19.139.190:37411)
0928 02:53:47.011889 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.011907 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.011958 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:53:47.012804 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.012837 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.013257 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.013624 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:47.014273 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.015706 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.016645 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.017321 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.017476 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.020057 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:47.020063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:47.020077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:47.020106 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:47.021950 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.022470 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.022475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.022480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.022602 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.022936 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.022939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.022940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.023367 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:47.023526 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.023839 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.024127 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260928 02:53:47.025068 2982 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:53:47.025358 2982 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:47.025411 2982 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda06403ce1e41bfa66dc70a25c26d05, State: Initialized, Role: FOLLOWER
I20260928 02:53:47.025840 2982 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:53:47.025969 2982 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:47.026064 2982 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:47.026173 2982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:47.027305 2982 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:53:47.027637 2982 leader_election.cc:302] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda06403ce1e41bfa66dc70a25c26d05; no voters:
I20260928 02:53:47.027947 2982 leader_election.cc:288] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:47.028106 2987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:47.028461 2987 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 LEADER]: Becoming Leader. State: Replica: cda06403ce1e41bfa66dc70a25c26d05, State: Running, Role: LEADER
I20260928 02:53:47.028829 2987 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:53:47.029167 2982 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:47.030390 2988 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda06403ce1e41bfa66dc70a25c26d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } } }
I20260928 02:53:47.030519 2988 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:47.030887 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda06403ce1e41bfa66dc70a25c26d05. Latest consensus state: current_term: 1 leader_uuid: "cda06403ce1e41bfa66dc70a25c26d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } } }
I20260928 02:53:47.030984 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:47.031280 2997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:47.034731 2997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:47.040977 2997 catalog_manager.cc:1347] Generated new cluster ID: 28861521ec64406abc26885d1a87a43a
I20260928 02:53:47.041070 2997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:47.063917 2997 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:47.064939 2997 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:47.072757 2997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Generated new TSK 0
I20260928 02:53:47.073465 2997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:53:47.108299 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37411
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:47.228884 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:47.229189 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:47.229236 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:47.233732 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:47.233817 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:47.233878 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:53:47.239686 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:47.240880 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:47.241932 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:47.251878 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:47.251888 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:47.251940 3008 server_base.cc:1034] running on GCE node
W20260928 02:53:47.251888 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:47.252560 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:47.253269 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:47.254456 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790564027254439 us; error 29 us; skew 500 ppm
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:47.257339 3008 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:53:47.258237 3008 webserver.cc:460] Webserver started at http://127.19.139.129:45461/ using document root <none> and password file <none>
I20260928 02:53:47.258773 3008 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:47.258832 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:47.258999 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:47.261062 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "873be534afce8e65474373aa91affdbc"
server_key_iv: "f729f60acf1c14704fe994d1ced88dfc"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.261557 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "873be534afce8e65474373aa91affdbc"
server_key_iv: "f729f60acf1c14704fe994d1ced88dfc"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.265237 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:47.267680 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.268980 3008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:53:47.269094 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "873be534afce8e65474373aa91affdbc"
server_key_iv: "f729f60acf1c14704fe994d1ced88dfc"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.269212 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:47.283917 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:47.287187 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:47.287374 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:47.287978 3008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:47.288877 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:47.288924 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.288977 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:47.289006 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.297508 3008 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:39269
I20260928 02:53:47.297535 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:39269 every 8 connection(s)
I20260928 02:53:47.298404 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:47.305469 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3008
I20260928 02:53:47.305604 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:47.305914 20014 external_mini_cluster.cc:1428] Setting key ad11cf1e85e4a44f6d695980bb85d796
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:47.312681 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.300235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:58359 (local address 127.19.139.190:37411)
0928 02:53:47.300366 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.300370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.301258 (+ 888us) server_negotiation.cc:407] Connection header received
0928 02:53:47.302554 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.302558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.302606 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.302680 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:47.303906 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.304496 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.305657 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.306088 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.306199 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.308806 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:47.308811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:47.308814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:47.308845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:47.310345 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.310961 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.310965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.310967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.311068 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.311473 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.311476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.311477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.311693 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:47.311785 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.312367 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.312506 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:53:47.313465 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.300628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:58359)
0928 02:53:47.301096 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.301140 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.302340 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.302907 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.302914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.303279 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:47.303765 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.303779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.304589 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.304591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.305515 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.305526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.306250 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.306254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.306458 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.307050 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:47.307066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:47.308606 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:47.310474 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:47.310480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:47.310496 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:47.310860 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:47.311167 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:47.311171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:47.311173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:47.311378 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:47.311806 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:47.311814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:47.312077 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.312287 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.312565 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260928 02:53:47.314759 3137 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37411
I20260928 02:53:47.315001 3137 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:53:47.315515 3137 heartbeater.cc:511] Master 127.19.139.190:37411 requested a full tablet report, sending...
I20260928 02:53:47.317063 2946 ts_manager.cc:194] Registered new tserver with Master: c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:53:47.318344 2946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:58359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:53:47.333292 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37411
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:47.442238 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:47.442528 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:47.442571 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:47.446593 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:47.446667 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:47.446727 3142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:53:47.452296 3142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:47.453438 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:47.454475 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:47.464289 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:47.464391 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:47.464452 3142 server_base.cc:1034] running on GCE node
W20260928 02:53:47.464391 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:47.465027 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:47.465643 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:47.466823 3142 hybrid_clock.cc:648] HybridClock initialized: now 1790564027466771 us; error 65 us; skew 500 ppm
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:47.469833 3142 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:53:47.470806 3142 webserver.cc:460] Webserver started at http://127.19.139.130:45769/ using document root <none> and password file <none>
I20260928 02:53:47.471376 3142 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:47.471441 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:47.471638 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:47.473996 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "50e7394af9a84c93b1d6124773b348a3"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "ab2d1fb7e59f4825ae70d3379a59a1b3"
server_key_iv: "4da4f6d8b90cdca1c93aaf3449a87a9f"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.474570 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "50e7394af9a84c93b1d6124773b348a3"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "ab2d1fb7e59f4825ae70d3379a59a1b3"
server_key_iv: "4da4f6d8b90cdca1c93aaf3449a87a9f"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.478648 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:53:47.481415 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.482702 3142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:53:47.482806 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "50e7394af9a84c93b1d6124773b348a3"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "ab2d1fb7e59f4825ae70d3379a59a1b3"
server_key_iv: "4da4f6d8b90cdca1c93aaf3449a87a9f"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.482959 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:47.495704 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:47.498957 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:47.499125 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:47.499737 3142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:47.500653 3142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:47.500715 3142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.500764 3142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:47.500794 3142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.509935 3142 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:39707
I20260928 02:53:47.509964 3270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:39707 every 8 connection(s)
I20260928 02:53:47.510874 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:47.519867 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3142
I20260928 02:53:47.520011 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:47.520350 20014 external_mini_cluster.cc:1428] Setting key 8107359dcfb5620f845af91db0738b99
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:47.524554 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.512632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:33319 (local address 127.19.139.190:37411)
0928 02:53:47.512778 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.512782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.513657 (+ 875us) server_negotiation.cc:407] Connection header received
0928 02:53:47.515000 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.515003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.515061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.515137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:47.516447 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.516992 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.518144 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.518370 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.518467 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.521157 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:47.521160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:47.521162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:47.521187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:47.522483 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.522977 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.522980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.522982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.523057 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.523392 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.523396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.523398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.523608 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:47.523696 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.524264 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.524393 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:53:47.525497 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.513040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:33319)
0928 02:53:47.513498 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.513543 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.514772 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.515343 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.515351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.515732 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:47.516286 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.516304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.517120 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.517122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.518024 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.518033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.518528 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.518533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.518699 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.519311 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:47.519328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:47.520996 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:47.522595 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:47.522601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:47.522618 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:47.522880 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:47.523143 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:47.523146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:47.523148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:47.523301 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:47.523711 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:47.523719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:47.523981 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.524197 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.524569 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260928 02:53:47.526894 3271 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37411
I20260928 02:53:47.527130 3271 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:47.527630 3271 heartbeater.cc:511] Master 127.19.139.190:37411 requested a full tablet report, sending...
I20260928 02:53:47.528822 2946 ts_manager.cc:194] Registered new tserver with Master: 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:53:47.529628 2946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:33319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:53:47.548028 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37411
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:47.660902 3276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:47.661182 3276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:47.661243 3276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:47.665802 3276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:47.665886 3276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:47.665977 3276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:53:47.672111 3276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:53:47.673362 3276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:47.674497 3276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:47.684836 3282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:47.684902 3281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:47.684902 3284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:47.684984 3276 server_base.cc:1034] running on GCE node
I20260928 02:53:47.685604 3276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:47.686269 3276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:47.687424 3276 hybrid_clock.cc:648] HybridClock initialized: now 1790564027687404 us; error 32 us; skew 500 ppm
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:47.690523 3276 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:53:47.691504 3276 webserver.cc:460] Webserver started at http://127.19.139.131:39923/ using document root <none> and password file <none>
I20260928 02:53:47.692029 3276 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:47.692086 3276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:47.692317 3276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:47.694384 3276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "88463cb9de6e4785bd7dba43d3fe767a"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "4b08e67341baf99527b8d5d0983b69e9"
server_key_iv: "fdc40d7623a479e85e21d176e5d3a9b8"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.694902 3276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "88463cb9de6e4785bd7dba43d3fe767a"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "4b08e67341baf99527b8d5d0983b69e9"
server_key_iv: "fdc40d7623a479e85e21d176e5d3a9b8"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.699232 3276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:53:47.701972 3291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.703378 3276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:47.703509 3276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "88463cb9de6e4785bd7dba43d3fe767a"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "4b08e67341baf99527b8d5d0983b69e9"
server_key_iv: "fdc40d7623a479e85e21d176e5d3a9b8"
server_key_version: "encryptionkey@0"
I20260928 02:53:47.703631 3276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:47.718348 3276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:47.721819 3276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:47.722013 3276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:47.722633 3276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:47.723546 3276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:47.723596 3276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.723650 3276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:47.723680 3276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:47.732774 3276 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46309
I20260928 02:53:47.732796 3404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46309 every 8 connection(s)
I20260928 02:53:47.733683 3276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:47.734644 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3276
I20260928 02:53:47.734757 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:47.735008 20014 external_mini_cluster.cc:1428] Setting key 6122cc596b90d3bf0d92fffab21143c3
Sep 28 02:53:47 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564027, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:53:47.747148 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.735676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:47513 (local address 127.19.139.190:37411)
0928 02:53:47.735869 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.735873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.736882 (+ 1009us) server_negotiation.cc:407] Connection header received
0928 02:53:47.738255 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.738258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.738333 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.738404 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:47.739601 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.740121 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.741248 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.741478 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.741563 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.743942 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:47.743946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:47.743949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:47.743975 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:47.745167 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.745648 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.745652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.745654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.745731 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.746053 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.746056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.746058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.746244 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:47.746344 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.746851 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.746981 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 02:53:47.748126 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.736216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:47513)
0928 02:53:47.736728 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.736785 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.738054 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.738595 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.738601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.738991 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:47.739461 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.739474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.740251 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.740255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.741126 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.741135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.741614 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.741618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.741787 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.742305 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:47.742321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:47.743774 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:47.745276 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:47.745282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:47.745299 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:47.745552 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:47.745823 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:47.745826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:47.745828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:47.745964 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:47.746349 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:47.746356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:47.746611 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.746782 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.747126 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260928 02:53:47.749330 3405 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37411
I20260928 02:53:47.749578 3405 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:47.750119 3405 heartbeater.cc:511] Master 127.19.139.190:37411 requested a full tablet report, sending...
I20260928 02:53:47.751209 2946 ts_manager.cc:194] Registered new tserver with Master: 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309)
I20260928 02:53:47.751777 2946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:47513
I20260928 02:53:47.761148 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:53:47.770648 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.763124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.19.139.190:37411)
0928 02:53:47.763285 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.763289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.763415 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:53:47.763519 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.763521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.763564 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.763628 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:47.764343 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.765001 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.766023 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.766251 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.766337 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.767685 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:47.767689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:47.767692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:47.767719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:47.768940 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.769325 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.769329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.769331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.769409 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:47.769653 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:47.769656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:47.769658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:47.769890 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:47.770004 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.770371 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.770488 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:53:47.773653 2946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:47.776897 2946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:47.791663 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.785702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.19.139.130:39707)
0928 02:53:47.787018 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.787029 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.787051 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:53:47.787123 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.787128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.787385 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.787533 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:47.787937 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.788723 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.790222 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.790938 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.791061 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.791204 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.791396 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.791504 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":131,"threads_started":1}
I20260928 02:53:47.791659 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.785408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39707 (local address 127.0.0.1:55476)
0928 02:53:47.786516 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.786571 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.786707 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.787701 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.787704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.787721 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:47.787827 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.787832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.788844 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.788848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.790101 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.790110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.791067 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.791071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.791186 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.791231 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.791351 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.791489 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":2,"spinlock_wait_cycles":6912,"thread_start_us":78,"threads_started":1}
I20260928 02:53:47.792016 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.785415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46309 (local address 127.0.0.1:49268)
0928 02:53:47.786966 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.786979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.787060 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.787628 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.787631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.787654 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:47.787850 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.787859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.789367 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.789371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.790578 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.790586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.791658 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.791661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.791740 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.791761 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.791828 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.791892 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"mutex_wait_us":508,"thread_start_us":85,"threads_started":1}
I20260928 02:53:47.792055 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.785790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:39269 (local address 127.0.0.1:55142)
0928 02:53:47.786517 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.786571 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.786707 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.787224 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.787230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.787303 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:47.787439 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.787446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.789147 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.789151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.790433 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.790444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.791480 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.791483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.791565 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.791590 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.791635 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.791701 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":42,"spinlock_wait_cycles":2560,"thread_start_us":80,"threads_started":1}
I20260928 02:53:47.792510 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.785557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.19.139.131:46309)
0928 02:53:47.786191 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.786204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.787134 (+ 930us) server_negotiation.cc:407] Connection header received
0928 02:53:47.787208 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.787212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.787384 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.787616 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:47.788026 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.789220 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.790684 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.791375 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.791481 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.791600 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.792282 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.792396 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":167,"threads_started":1}
I20260928 02:53:47.794716 3205 tablet_service.cc:1467] Processing CreateTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 (DEFAULT_TABLE table=test-table [id=5a21b6aa6a8c4f2e9e576fc56cd901ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:47.794780 3339 tablet_service.cc:1467] Processing CreateTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 (DEFAULT_TABLE table=test-table [id=5a21b6aa6a8c4f2e9e576fc56cd901ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:47.795642 3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd39bb6c37f4dab9543c4b09c5748b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:47.795652 3339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd39bb6c37f4dab9543c4b09c5748b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:47.799649 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.786260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.19.139.129:39269)
0928 02:53:47.786672 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.786688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.786712 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:53:47.786840 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.786846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.787079 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.787234 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:47.787527 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.788991 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.790564 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.791353 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.792372 (+ 1019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.792498 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.792546 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.792630 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":125,"threads_started":1}
I20260928 02:53:47.802964 3423 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Bootstrap starting.
I20260928 02:53:47.802969 3071 tablet_service.cc:1467] Processing CreateTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 (DEFAULT_TABLE table=test-table [id=5a21b6aa6a8c4f2e9e576fc56cd901ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:47.803537 3422 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Bootstrap starting.
I20260928 02:53:47.804144 3071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd39bb6c37f4dab9543c4b09c5748b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:47.805635 3423 tablet_bootstrap.cc:654] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:47.805732 3422 tablet_bootstrap.cc:654] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:47.806449 3423 log.cc:824] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:47.807144 3422 log.cc:824] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:47.809201 3423 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: No bootstrap required, opened a new log
I20260928 02:53:47.809371 3422 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: No bootstrap required, opened a new log
I20260928 02:53:47.809427 3423 ts_tablet_manager.cc:1397] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:53:47.809579 3422 ts_tablet_manager.cc:1397] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:53:47.812490 3426 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap starting.
I20260928 02:53:47.812500 3423 raft_consensus.cc:348] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.812937 3423 raft_consensus.cc:374] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:47.813047 3423 raft_consensus.cc:729] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7394af9a84c93b1d6124773b348a3, State: Initialized, Role: FOLLOWER
I20260928 02:53:47.813355 3422 raft_consensus.cc:348] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.813594 3422 raft_consensus.cc:374] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:47.813666 3422 raft_consensus.cc:729] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88463cb9de6e4785bd7dba43d3fe767a, State: Initialized, Role: FOLLOWER
I20260928 02:53:47.813656 3423 consensus_queue.cc:260] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.814148 3422 consensus_queue.cc:260] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.814503 3271 heartbeater.cc:503] Master 127.19.139.190:37411 was elected leader, sending a full tablet report...
I20260928 02:53:47.815182 3422 ts_tablet_manager.cc:1428] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260928 02:53:47.815295 3426 tablet_bootstrap.cc:654] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:47.815654 3405 heartbeater.cc:503] Master 127.19.139.190:37411 was elected leader, sending a full tablet report...
I20260928 02:53:47.816256 3426 log.cc:824] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:47.816303 3423 ts_tablet_manager.cc:1428] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:53:47.818643 3426 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: No bootstrap required, opened a new log
I20260928 02:53:47.818846 3426 ts_tablet_manager.cc:1397] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:53:47.822779 3426 raft_consensus.cc:348] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.823069 3426 raft_consensus.cc:374] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:47.823117 3426 raft_consensus.cc:729] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79c4608e21a4ea58ad771a1b8c951ca, State: Initialized, Role: FOLLOWER
I20260928 02:53:47.823583 3426 consensus_queue.cc:260] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.824414 3137 heartbeater.cc:503] Master 127.19.139.190:37411 was elected leader, sending a full tablet report...
I20260928 02:53:47.824883 3426 ts_tablet_manager.cc:1428] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:53:47.870417 3428 raft_consensus.cc:482] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:47.870635 3428 raft_consensus.cc:504] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.871801 3428 leader_election.cc:288] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707), c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269)
I20260928 02:53:47.876019 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.872246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:43551 (local address 127.19.139.129:39269)
0928 02:53:47.872364 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.872369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.872410 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:53:47.872592 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.872596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.872653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.872734 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:47.873097 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.873833 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.875034 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.875431 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.875534 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.875587 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.875806 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.875886 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:53:47.876019 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.872235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:53705 (local address 127.19.139.130:39707)
0928 02:53:47.872372 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.872376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.872761 (+ 385us) server_negotiation.cc:407] Connection header received
0928 02:53:47.872828 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.872831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.872885 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.872984 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:47.873224 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.873795 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.875059 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.875499 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.875614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.875663 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.875827 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.875900 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:53:47.876019 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.872085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:39269 (local address 127.19.139.131:43551)
0928 02:53:47.872334 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.872349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.872472 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.872791 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.872794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.872818 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:47.872976 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.872984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.873957 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.873960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.874897 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.874908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.875558 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.875562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.875679 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.875705 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.875766 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.875854 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":91}
I20260928 02:53:47.876042 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.872105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39707 (local address 127.19.139.131:53705)
0928 02:53:47.872604 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.872617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:47.872749 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:47.872989 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:47.872991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:47.873003 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:47.873114 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.873121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.873913 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.873916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.874909 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:47.874919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.875627 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.875630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:47.875710 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.875729 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:53:47.875798 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:53:47.875870 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":67,"thread_start_us":173,"threads_started":1}
I20260928 02:53:47.876782 3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd39bb6c37f4dab9543c4b09c5748b0" candidate_uuid: "88463cb9de6e4785bd7dba43d3fe767a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7394af9a84c93b1d6124773b348a3" is_pre_election: true
I20260928 02:53:47.876843 3091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd39bb6c37f4dab9543c4b09c5748b0" candidate_uuid: "88463cb9de6e4785bd7dba43d3fe767a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" is_pre_election: true
I20260928 02:53:47.877108 3091 raft_consensus.cc:2454] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88463cb9de6e4785bd7dba43d3fe767a in term 0.
I20260928 02:53:47.877110 3225 raft_consensus.cc:2454] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88463cb9de6e4785bd7dba43d3fe767a in term 0.
I20260928 02:53:47.877622 3292 leader_election.cc:302] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a, c79c4608e21a4ea58ad771a1b8c951ca; no voters:
I20260928 02:53:47.877897 3428 raft_consensus.cc:2781] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:47.877985 3428 raft_consensus.cc:482] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:47.878028 3428 raft_consensus.cc:3037] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:47.879106 3428 raft_consensus.cc:504] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.879664 3428 leader_election.cc:288] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [CANDIDATE]: Term 1 election: Requested vote from peers 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707), c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269)
I20260928 02:53:47.880149 3091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd39bb6c37f4dab9543c4b09c5748b0" candidate_uuid: "88463cb9de6e4785bd7dba43d3fe767a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
I20260928 02:53:47.880149 3225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd39bb6c37f4dab9543c4b09c5748b0" candidate_uuid: "88463cb9de6e4785bd7dba43d3fe767a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7394af9a84c93b1d6124773b348a3"
I20260928 02:53:47.880338 3091 raft_consensus.cc:3037] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:47.880338 3225 raft_consensus.cc:3037] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:47.881713 3091 raft_consensus.cc:2454] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88463cb9de6e4785bd7dba43d3fe767a in term 1.
I20260928 02:53:47.881713 3225 raft_consensus.cc:2454] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88463cb9de6e4785bd7dba43d3fe767a in term 1.
I20260928 02:53:47.882081 3292 leader_election.cc:302] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a, c79c4608e21a4ea58ad771a1b8c951ca; no voters:
I20260928 02:53:47.882279 3428 raft_consensus.cc:2781] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:47.882565 3428 raft_consensus.cc:686] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 LEADER]: Becoming Leader. State: Replica: 88463cb9de6e4785bd7dba43d3fe767a, State: Running, Role: LEADER
I20260928 02:53:47.882951 3428 consensus_queue.cc:237] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:53:47.886651 2946 catalog_manager.cc:5495] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a reported cstate change: term changed from 0 to 1, leader changed from <none> to 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131). New cstate: current_term: 1 leader_uuid: "88463cb9de6e4785bd7dba43d3fe767a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } health_report { overall_health: UNKNOWN } } }
I20260928 02:53:47.915619 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.912099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.19.139.131:46309)
0928 02:53:47.912262 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:47.912266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:47.912280 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:53:47.912398 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:47.912401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:47.912453 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:47.912541 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:47.912791 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.913293 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.914196 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:47.914422 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:47.914508 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:47.914810 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:47.915241 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:47.915361 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:47.915423 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:53:47.915479 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:53:47.921591 3091 raft_consensus.cc:1261] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Refusing update from remote peer 88463cb9de6e4785bd7dba43d3fe767a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:47.921591 3225 raft_consensus.cc:1261] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Refusing update from remote peer 88463cb9de6e4785bd7dba43d3fe767a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:47.922324 3432 consensus_queue.cc:1035] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:47.922500 3428 consensus_queue.cc:1035] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:47.939241 2945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:47.939455 2945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:47.942294 2945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Sending DeleteTablet for 3 replicas of tablet dbd39bb6c37f4dab9543c4b09c5748b0
I20260928 02:53:47.943187 3339 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:49268
I20260928 02:53:47.943290 3205 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:55476
I20260928 02:53:47.943363 3071 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:55142
I20260928 02:53:47.943782 3445 tablet_replica.cc:330] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: stopping tablet replica
I20260928 02:53:47.944134 3445 raft_consensus.cc:2229] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:47.944175 3446 tablet_replica.cc:330] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: stopping tablet replica
I20260928 02:53:47.944149 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3008
I20260928 02:53:47.944603 3446 raft_consensus.cc:2229] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:47.944954 3446 raft_consensus.cc:2258] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:47.945812 3444 tablet_replica.cc:330] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: stopping tablet replica
I20260928 02:53:47.946135 3444 raft_consensus.cc:2229] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:47.946609 3444 raft_consensus.cc:2258] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:47.946748 3446 ts_tablet_manager.cc:1905] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:47.948098 3444 ts_tablet_manager.cc:1905] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:47.949610 3446 ts_tablet_manager.cc:1918] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:47.949710 3446 log.cc:1196] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/dbd39bb6c37f4dab9543c4b09c5748b0
I20260928 02:53:47.950009 3446 ts_tablet_manager.cc:1939] T dbd39bb6c37f4dab9543c4b09c5748b0 P 50e7394af9a84c93b1d6124773b348a3: Deleting consensus metadata
I20260928 02:53:47.951062 3444 ts_tablet_manager.cc:1918] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:47.951244 3444 log.cc:1196] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/dbd39bb6c37f4dab9543c4b09c5748b0
I20260928 02:53:47.951649 3444 ts_tablet_manager.cc:1939] T dbd39bb6c37f4dab9543c4b09c5748b0 P 88463cb9de6e4785bd7dba43d3fe767a: Deleting consensus metadata
I20260928 02:53:47.951658 2930 catalog_manager.cc:4841] TS 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707): tablet dbd39bb6c37f4dab9543c4b09c5748b0 (table test-table [id=5a21b6aa6a8c4f2e9e576fc56cd901ad]) successfully deleted
W20260928 02:53:47.952205 2933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:39269 (error 108)
I20260928 02:53:47.952421 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3142
I20260928 02:53:47.953619 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:47.953288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:39269 (local address 127.0.0.1:55152)
0928 02:53:47.953456 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:47.953529 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:53:47.953758 2933 catalog_manager.cc:4584] TS c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd39bb6c37f4dab9543c4b09c5748b0: Network error: Client connection negotiation failed: client connection to 127.19.139.129:39269: connect: Connection refused (error 111)
I20260928 02:53:47.956993 2931 catalog_manager.cc:4841] TS 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309): tablet dbd39bb6c37f4dab9543c4b09c5748b0 (table test-table [id=5a21b6aa6a8c4f2e9e576fc56cd901ad]) successfully deleted
I20260928 02:53:47.958514 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3276
I20260928 02:53:47.963440 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 2914
I20260928 02:54:07.969017 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37411
--webserver_interface=127.19.139.190
--webserver_port=34651
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:08.081686 3448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:08.081979 3448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:08.082018 3448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:08.086285 3448 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:54:08.086359 3448 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:54:08.086374 3448 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:54:08.086387 3448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:08.086405 3448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:08.086422 3448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:08.086437 3448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:08.086460 3448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:08.092063 3448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:08.093230 3448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:08.094236 3448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:08.107858 3448 server_base.cc:1034] running on GCE node
W20260928 02:54:08.107779 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.107914 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.107781 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:08.108795 3448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:08.110098 3448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:08.111282 3448 hybrid_clock.cc:648] HybridClock initialized: now 1790564048111262 us; error 37 us; skew 500 ppm
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:08.114270 3448 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:54:08.115309 3448 webserver.cc:460] Webserver started at http://127.19.139.190:34651/ using document root <none> and password file <none>
I20260928 02:54:08.115945 3448 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:08.116037 3448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:08.120388 3448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:08.122809 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.124132 3448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:54:08.124311 3448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "cda06403ce1e41bfa66dc70a25c26d05"
format_stamp: "Formatted at 2026-09-28 02:53:46 on dist-test-slave-5vc0"
server_key: "8171132a6791ae481bafbd0629aa3413"
server_key_iv: "0fa154c74e7d1137b38671af0386122f"
server_key_version: "encryptionkey@0"
I20260928 02:54:08.124987 3448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:08.142664 3448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:08.145983 3448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:08.146189 3448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:08.153327 3448 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:37411
I20260928 02:54:08.153589 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:37411 every 8 connection(s)
I20260928 02:54:08.154515 3448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:08.155833 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3448
I20260928 02:54:08.156391 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:39269
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:37411
--webserver_port=45461
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:54:08.159955 3517 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:54:08.161900 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap starting.
I20260928 02:54:08.179164 3517 log.cc:824] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:08.188328 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:54:08.188701 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap complete.
I20260928 02:54:08.192806 3517 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:08.193251 3517 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda06403ce1e41bfa66dc70a25c26d05, State: Initialized, Role: FOLLOWER
I20260928 02:54:08.193773 3517 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:08.193908 3517 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:08.193980 3517 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:08.194073 3517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:54:08.196146 3517 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:08.196558 3517 leader_election.cc:302] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda06403ce1e41bfa66dc70a25c26d05; no voters:
I20260928 02:54:08.196892 3517 leader_election.cc:288] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:54:08.197010 3521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:54:08.197216 3521 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 LEADER]: Becoming Leader. State: Replica: cda06403ce1e41bfa66dc70a25c26d05, State: Running, Role: LEADER
I20260928 02:54:08.197587 3521 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:08.198222 3517 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:08.199156 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda06403ce1e41bfa66dc70a25c26d05. Latest consensus state: current_term: 2 leader_uuid: "cda06403ce1e41bfa66dc70a25c26d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } } }
I20260928 02:54:08.199306 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:08.199702 3530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:08.199753 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cda06403ce1e41bfa66dc70a25c26d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } } }
I20260928 02:54:08.199854 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:08.204977 3530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:08.205706 3530 catalog_manager.cc:1259] Loaded cluster ID: 28861521ec64406abc26885d1a87a43a
I20260928 02:54:08.205766 3530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:08.207564 3530 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:08.208216 3530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Loaded TSK: 0
I20260928 02:54:08.224802 3530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Generated new TSK 1
I20260928 02:54:08.225108 3530 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:54:08.282367 3519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:08.282650 3519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:08.282699 3519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:08.286928 3519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:08.287006 3519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:08.287075 3519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:08.292604 3519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:39269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=45461
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:08.293721 3519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:08.294801 3519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:08.304580 3545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.304589 3544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.304563 3547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:08.304927 3519 server_base.cc:1034] running on GCE node
I20260928 02:54:08.305327 3519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:08.305990 3519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:08.307195 3519 hybrid_clock.cc:648] HybridClock initialized: now 1790564048307171 us; error 34 us; skew 500 ppm
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:08.313511 3519 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:54:08.314524 3519 webserver.cc:460] Webserver started at http://127.19.139.129:45461/ using document root <none> and password file <none>
I20260928 02:54:08.315102 3519 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:08.315179 3519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:08.319497 3519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:54:08.322060 3554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.323639 3519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:54:08.323773 3519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "873be534afce8e65474373aa91affdbc"
server_key_iv: "f729f60acf1c14704fe994d1ced88dfc"
server_key_version: "encryptionkey@0"
I20260928 02:54:08.324450 3519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:08.335117 3519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:08.338402 3519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:08.338585 3519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:08.339186 3519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:08.340623 3561 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:54:08.346205 3519 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:54:08.346287 3519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 02:54:08.346351 3519 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:54:08.348106 3519 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:54:08.348181 3519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 02:54:08.348501 3561 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap starting.
I20260928 02:54:08.357909 3519 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:39269
I20260928 02:54:08.357990 3668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:39269 every 8 connection(s)
I20260928 02:54:08.358932 3519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:08.363591 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3519
I20260928 02:54:08.364081 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:39707
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:37411
--webserver_port=45769
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:54:08.368738 3561 log.cc:824] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:08.377166 3561 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:54:08.377578 3561 tablet_bootstrap.cc:492] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap complete.
I20260928 02:54:08.378818 3561 ts_tablet_manager.cc:1397] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260928 02:54:08.379024 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.361315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:60815 (local address 127.19.139.190:37411)
0928 02:54:08.361791 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.361813 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.362229 (+ 416us) server_negotiation.cc:407] Connection header received
0928 02:54:08.363765 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.363781 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.364262 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.364655 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:08.366344 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.367432 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.369088 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.369859 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.370047 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.373114 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:08.373121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:08.373137 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:08.373173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:08.375278 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.375966 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.375973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.375978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.376143 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.376570 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.376574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.376576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.377059 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:08.377196 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.377852 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.378167 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
I20260928 02:54:08.379029 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.361303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:60815)
0928 02:54:08.361993 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.362075 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.363229 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.364783 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.364795 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.365395 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:08.366114 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.366143 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.367598 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.367602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.368950 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.368962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.370050 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.370054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.370239 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.370928 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:08.370948 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:08.372877 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:08.375422 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.375429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.375444 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.375835 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.376276 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.376280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.376282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.376474 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.377170 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:08.377176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:08.377464 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.377653 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.377976 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":128,"threads_started":1}
I20260928 02:54:08.382021 3669 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37411
I20260928 02:54:08.381671 3561 raft_consensus.cc:348] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.382215 3561 raft_consensus.cc:729] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79c4608e21a4ea58ad771a1b8c951ca, State: Initialized, Role: FOLLOWER
I20260928 02:54:08.382263 3669 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:08.382733 3561 consensus_queue.cc:260] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.382839 3669 heartbeater.cc:511] Master 127.19.139.190:37411 requested a full tablet report, sending...
I20260928 02:54:08.383678 3561 ts_tablet_manager.cc:1428] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:54:08.385425 3481 ts_manager.cc:194] Registered new tserver with Master: c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269)
I20260928 02:54:08.388808 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:60815
I20260928 02:54:08.391429 3669 heartbeater.cc:503] Master 127.19.139.190:37411 was elected leader, sending a full tablet report...
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.129@KRBTEST.COM
I20260928 02:54:08.398404 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.388275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:39269 (local address 127.0.0.1:56018)
0928 02:54:08.389040 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.389083 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.389223 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.389723 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.389729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.390095 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:08.390200 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.390207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.391142 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.391145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.391846 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.391858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.392608 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.392612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.392718 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.393254 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:08.393266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:08.394777 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:08.396723 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.396731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.396736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.396965 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.397242 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.397244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.397246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.397319 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.397792 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:08.397797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:08.397924 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.398065 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.398260 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":124,"threads_started":1}
I20260928 02:54:08.398550 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.388365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.19.139.129:39269)
0928 02:54:08.388788 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.388799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.389183 (+ 384us) server_negotiation.cc:407] Connection header received
0928 02:54:08.389358 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.389363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.389577 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.389704 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:08.390321 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.391031 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.391959 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.392483 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.392587 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.394918 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:08.394927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:08.394939 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:08.394964 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:08.396627 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.397064 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.397071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.397072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.397162 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.397391 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.397394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.397396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.397678 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:08.397798 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.398148 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.398321 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":120,"threads_started":1}
I20260928 02:54:08.399567 3603 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd39bb6c37f4dab9543c4b09c5748b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:56018
I20260928 02:54:08.400610 3678 tablet_replica.cc:330] stopping tablet replica
I20260928 02:54:08.400872 3678 raft_consensus.cc:2229] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:08.401105 3678 raft_consensus.cc:2258] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:08.402038 3678 ts_tablet_manager.cc:1905] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:08.405588 3678 ts_tablet_manager.cc:1918] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:08.405689 3678 log.cc:1196] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal/wals/dbd39bb6c37f4dab9543c4b09c5748b0
I20260928 02:54:08.405915 3678 ts_tablet_manager.cc:1939] T dbd39bb6c37f4dab9543c4b09c5748b0 P c79c4608e21a4ea58ad771a1b8c951ca: Deleting consensus metadata
I20260928 02:54:08.407831 3468 catalog_manager.cc:4841] TS c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269): tablet dbd39bb6c37f4dab9543c4b09c5748b0 (table test-table [id=5a21b6aa6a8c4f2e9e576fc56cd901ad]) successfully deleted
W20260928 02:54:08.484100 3673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:08.484416 3673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:08.484460 3673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:08.488588 3673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:08.488657 3673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:08.488718 3673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:08.494127 3673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:39707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=45769
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:08.495201 3673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:08.496234 3673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:08.506134 3673 server_base.cc:1034] running on GCE node
W20260928 02:54:08.506131 3684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.506166 3683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.506170 3686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:08.506791 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:08.507452 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:08.508611 3673 hybrid_clock.cc:648] HybridClock initialized: now 1790564048508577 us; error 46 us; skew 500 ppm
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:08.511371 3673 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:54:08.512399 3673 webserver.cc:460] Webserver started at http://127.19.139.130:45769/ using document root <none> and password file <none>
I20260928 02:54:08.512969 3673 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:08.513051 3673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:08.517102 3673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:08.519249 3693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.520576 3673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:08.520728 3673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "50e7394af9a84c93b1d6124773b348a3"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "ab2d1fb7e59f4825ae70d3379a59a1b3"
server_key_iv: "4da4f6d8b90cdca1c93aaf3449a87a9f"
server_key_version: "encryptionkey@0"
I20260928 02:54:08.521281 3673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:08.549353 3673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:08.552699 3673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:08.552866 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:08.553416 3673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:08.554209 3673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:08.554263 3673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.554302 3673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:08.554323 3673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.562968 3673 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:39707
I20260928 02:54:08.563006 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:39707 every 8 connection(s)
I20260928 02:54:08.563949 3673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:08.572232 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3673
I20260928 02:54:08.572645 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:46309
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:37411
--webserver_port=39923
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:08.578027 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.565700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:51739 (local address 127.19.139.190:37411)
0928 02:54:08.565824 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.565827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.566700 (+ 873us) server_negotiation.cc:407] Connection header received
0928 02:54:08.568027 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.568030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.568081 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.568178 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:08.569383 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.569928 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.571064 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.571348 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.571443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.574271 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:08.574275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:08.574277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:08.574303 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:08.575670 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.576349 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.576353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.576354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.576456 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.576859 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.576861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.576863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.577057 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:08.577128 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.577782 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.577902 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:54:08.579205 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.566059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:51739)
0928 02:54:08.566537 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.566579 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.567834 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.568341 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.568347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.568703 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:08.569229 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.569243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.570047 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.570050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.570936 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.570945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.571494 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.571497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.571640 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.572367 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:08.572388 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:08.574079 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:08.575838 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.575845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.575860 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.576253 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.576544 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.576548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.576550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.576754 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.577150 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:08.577156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:08.577499 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.577714 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.578094 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260928 02:54:08.580495 3807 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37411
I20260928 02:54:08.580770 3807 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:08.581266 3807 heartbeater.cc:511] Master 127.19.139.190:37411 requested a full tablet report, sending...
I20260928 02:54:08.582342 3481 ts_manager.cc:194] Registered new tserver with Master: 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
I20260928 02:54:08.582907 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:51739
W20260928 02:54:08.685705 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:08.685985 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:08.686048 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:08.690163 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:08.690229 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:08.690308 3810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:08.695603 3810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:46309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=39923
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:08.696739 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:08.697755 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:08.707680 3815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.707710 3818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:08.707682 3816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:08.707903 3810 server_base.cc:1034] running on GCE node
I20260928 02:54:08.708500 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:08.709163 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:08.710345 3810 hybrid_clock.cc:648] HybridClock initialized: now 1790564048710324 us; error 33 us; skew 500 ppm
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:08.713395 3810 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:54:08.714300 3810 webserver.cc:460] Webserver started at http://127.19.139.131:39923/ using document root <none> and password file <none>
I20260928 02:54:08.714840 3810 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:08.714911 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:08.719082 3810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:54:08.721356 3825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.722805 3810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:08.722926 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "88463cb9de6e4785bd7dba43d3fe767a"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "4b08e67341baf99527b8d5d0983b69e9"
server_key_iv: "fdc40d7623a479e85e21d176e5d3a9b8"
server_key_version: "encryptionkey@0"
I20260928 02:54:08.723541 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:08.758112 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:08.761488 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:08.761713 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:08.762313 3810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:08.763164 3810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:08.763211 3810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.763250 3810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:08.763278 3810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:08.771461 3810 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46309
I20260928 02:54:08.771492 3938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46309 every 8 connection(s)
I20260928 02:54:08.772480 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:08.779822 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3810
Sep 28 02:54:08 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564048, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:08.786896 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.774305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:45193 (local address 127.19.139.190:37411)
0928 02:54:08.774492 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.774497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.775297 (+ 800us) server_negotiation.cc:407] Connection header received
0928 02:54:08.776622 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.776625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.776680 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.776779 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:08.777977 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.778533 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.779677 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.779996 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.780107 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.783085 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:08.783089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:08.783092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:08.783119 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:08.784526 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.785036 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.785040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.785041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.785125 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.785467 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.785470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.785472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.785675 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:08.785760 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.786425 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.786770 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:54:08.787740 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.774689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:45193)
0928 02:54:08.775144 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.775186 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.776384 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.776979 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.776986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.777344 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:08.777815 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.777829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.778639 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.778642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.779548 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.779557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.780142 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.780145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.780348 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.781056 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:08.781076 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:08.782927 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:08.784642 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.784647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.784659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.784942 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.785231 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:08.785234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:08.785235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:08.785377 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:08.785777 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:08.785784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:08.786140 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.786355 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.786718 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260928 02:54:08.788893 3939 heartbeater.cc:347] Connected to a master server at 127.19.139.190:37411
I20260928 02:54:08.789002 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.780608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.19.139.190:37411)
0928 02:54:08.780883 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.780887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.781003 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:54:08.781122 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.781124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.781162 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.781230 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:08.782102 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.782915 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.784024 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.784274 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.784357 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.785766 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:08.785770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:08.785773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:08.785798 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:08.787313 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.787699 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.787702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.787704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.787793 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.788204 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.788208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.788210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.788377 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:08.788568 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.788721 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.788833 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20260928 02:54:08.789144 3939 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:08.789618 3939 heartbeater.cc:511] Master 127.19.139.190:37411 requested a full tablet report, sending...
I20260928 02:54:08.790552 3481 ts_manager.cc:194] Registered new tserver with Master: 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309)
I20260928 02:54:08.791164 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:45193
I20260928 02:54:08.791676 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:08.795511 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.792257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.19.139.190:37411)
0928 02:54:08.792355 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.792358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.792610 (+ 252us) server_negotiation.cc:407] Connection header received
0928 02:54:08.792764 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.792766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.792854 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.792934 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:08.793165 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.793653 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.794465 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.794680 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.794765 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.795035 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:08.795153 (+ 118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:54:08.795379 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56192: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:54:08.795570 3943 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56192: token expired
W20260928 02:54:08.795871 3408 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.139.190:37411)
I20260928 02:54:08.802352 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.796258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.19.139.190:37411)
0928 02:54:08.796356 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.796359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.796417 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:54:08.796504 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.796506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.796538 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.796595 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:08.797089 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.797549 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.798293 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.798544 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.798613 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.799799 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:08.799803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:08.799805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:08.799826 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:08.801129 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.801493 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.801496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.801497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.801564 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:08.801741 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:08.801742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:08.801743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:08.801893 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:08.801985 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.802147 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.802235 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:54:08.804884 3480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:08.808130 3480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:08.816576 3603 tablet_service.cc:1467] Processing CreateTablet for tablet 55e342e37efe4d939f3eee09605ec684 (DEFAULT_TABLE table=test-table [id=1325be8c6092483c8b61098a092e8baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:08.817283 3603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e342e37efe4d939f3eee09605ec684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:08.820065 3561 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap starting.
I20260928 02:54:08.821020 3561 tablet_bootstrap.cc:654] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:08.822139 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.815701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39707 (local address 127.0.0.1:49832)
0928 02:54:08.816498 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.816510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.816602 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.817105 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.817108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.817127 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:08.817243 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.817250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.818212 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.818216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.819994 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.820004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.821490 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.821494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.821597 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.821653 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.821727 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.821977 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":261,"thread_start_us":130,"threads_started":1}
I20260928 02:54:08.822139 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.815164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46309 (local address 127.0.0.1:42324)
0928 02:54:08.815319 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.815336 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.815437 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.816690 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.816693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.816712 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:08.816828 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.816834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.818333 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.818336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.819517 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.819527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.821785 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.821789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.821905 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.821929 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.821974 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.822046 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":32}
I20260928 02:54:08.822746 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.815345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.19.139.131:46309)
0928 02:54:08.815813 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.815826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.815853 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:54:08.815987 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.815994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.816506 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.817028 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:08.817193 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.818227 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.819652 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.821665 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.822338 (+ 673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.822483 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.822530 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.822612 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":213,"threads_started":1}
I20260928 02:54:08.822752 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.815974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.19.139.130:39707)
0928 02:54:08.816309 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.816321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.816597 (+ 276us) server_negotiation.cc:407] Connection header received
0928 02:54:08.816709 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.816714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.816919 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.817146 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:08.817321 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.818102 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.820258 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.821122 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.821252 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.821419 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.822484 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.822603 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260928 02:54:08.822840 3561 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: No bootstrap required, opened a new log
I20260928 02:54:08.822937 3561 ts_tablet_manager.cc:1397] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:54:08.823351 3561 raft_consensus.cc:348] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.823458 3561 raft_consensus.cc:374] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:08.823496 3561 raft_consensus.cc:729] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79c4608e21a4ea58ad771a1b8c951ca, State: Initialized, Role: FOLLOWER
I20260928 02:54:08.823652 3561 consensus_queue.cc:260] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.823915 3561 ts_tablet_manager.cc:1428] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:54:08.826022 3741 tablet_service.cc:1467] Processing CreateTablet for tablet 55e342e37efe4d939f3eee09605ec684 (DEFAULT_TABLE table=test-table [id=1325be8c6092483c8b61098a092e8baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:08.826380 3873 tablet_service.cc:1467] Processing CreateTablet for tablet 55e342e37efe4d939f3eee09605ec684 (DEFAULT_TABLE table=test-table [id=1325be8c6092483c8b61098a092e8baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:08.827152 3741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e342e37efe4d939f3eee09605ec684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:08.827406 3873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e342e37efe4d939f3eee09605ec684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:08.833917 3952 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Bootstrap starting.
I20260928 02:54:08.833917 3953 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Bootstrap starting.
I20260928 02:54:08.836298 3952 tablet_bootstrap.cc:654] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:08.836309 3953 tablet_bootstrap.cc:654] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:08.837059 3952 log.cc:824] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:08.837049 3953 log.cc:824] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:08.839136 3953 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: No bootstrap required, opened a new log
I20260928 02:54:08.839136 3952 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: No bootstrap required, opened a new log
I20260928 02:54:08.839334 3952 ts_tablet_manager.cc:1397] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 02:54:08.839334 3953 ts_tablet_manager.cc:1397] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 02:54:08.843298 3953 raft_consensus.cc:348] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.843298 3952 raft_consensus.cc:348] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.843619 3953 raft_consensus.cc:374] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:08.843619 3952 raft_consensus.cc:374] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:08.843679 3953 raft_consensus.cc:729] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88463cb9de6e4785bd7dba43d3fe767a, State: Initialized, Role: FOLLOWER
I20260928 02:54:08.843679 3952 raft_consensus.cc:729] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7394af9a84c93b1d6124773b348a3, State: Initialized, Role: FOLLOWER
I20260928 02:54:08.844220 3953 consensus_queue.cc:260] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.844199 3952 consensus_queue.cc:260] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.844935 3807 heartbeater.cc:503] Master 127.19.139.190:37411 was elected leader, sending a full tablet report...
I20260928 02:54:08.845309 3952 ts_tablet_manager.cc:1428] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:54:08.845309 3953 ts_tablet_manager.cc:1428] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:54:08.845990 3939 heartbeater.cc:503] Master 127.19.139.190:37411 was elected leader, sending a full tablet report...
W20260928 02:54:08.860222 3670 tablet.cc:2369] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:08.904505 3956 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:08.904721 3956 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.905839 3956 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269)
I20260928 02:54:08.910989 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.906240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:41225 (local address 127.19.139.131:46309)
0928 02:54:08.906370 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.906375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.906645 (+ 270us) server_negotiation.cc:407] Connection header received
0928 02:54:08.906729 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.906732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.906776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.906846 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:08.907051 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.907838 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.909665 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.910289 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.910431 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.910497 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.910747 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.910830 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:54:08.910992 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.906210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:36027 (local address 127.19.139.129:39269)
0928 02:54:08.906508 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.906512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.906525 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:54:08.906571 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.906573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.906630 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.906738 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:08.907009 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.907838 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.909665 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.910343 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.910455 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.910611 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.910789 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.910863 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260928 02:54:08.910991 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.906101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:39269 (local address 127.19.139.130:36027)
0928 02:54:08.906306 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.906321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.906423 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.906761 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.906763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.906777 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:08.906895 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.906902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.907971 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.907975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.909518 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.909530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.910440 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.910443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.910604 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.910642 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.910767 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.910825 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":161}
I20260928 02:54:08.911000 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.906101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46309 (local address 127.19.139.130:41225)
0928 02:54:08.906540 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.906552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:08.906651 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:08.906859 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:08.906861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:08.906874 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:08.906962 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.906968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.907971 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.907975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.909517 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:08.909530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.910433 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.910438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:08.910611 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.910641 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:54:08.910720 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:54:08.910809 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":227,"thread_start_us":113,"threads_started":1}
I20260928 02:54:08.911736 3893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e342e37efe4d939f3eee09605ec684" candidate_uuid: "50e7394af9a84c93b1d6124773b348a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88463cb9de6e4785bd7dba43d3fe767a" is_pre_election: true
I20260928 02:54:08.911720 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e342e37efe4d939f3eee09605ec684" candidate_uuid: "50e7394af9a84c93b1d6124773b348a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" is_pre_election: true
I20260928 02:54:08.912027 3623 raft_consensus.cc:2454] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e7394af9a84c93b1d6124773b348a3 in term 0.
I20260928 02:54:08.912034 3893 raft_consensus.cc:2454] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e7394af9a84c93b1d6124773b348a3 in term 0.
I20260928 02:54:08.912549 3697 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e7394af9a84c93b1d6124773b348a3, c79c4608e21a4ea58ad771a1b8c951ca; no voters:
I20260928 02:54:08.912870 3956 raft_consensus.cc:2781] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:08.912952 3956 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:08.912984 3956 raft_consensus.cc:3037] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:08.914042 3956 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.914430 3956 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [CANDIDATE]: Term 1 election: Requested vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269)
I20260928 02:54:08.914789 3893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e342e37efe4d939f3eee09605ec684" candidate_uuid: "50e7394af9a84c93b1d6124773b348a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88463cb9de6e4785bd7dba43d3fe767a"
I20260928 02:54:08.914861 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e342e37efe4d939f3eee09605ec684" candidate_uuid: "50e7394af9a84c93b1d6124773b348a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
I20260928 02:54:08.914927 3893 raft_consensus.cc:3037] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:08.914987 3623 raft_consensus.cc:3037] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:08.916234 3623 raft_consensus.cc:2454] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e7394af9a84c93b1d6124773b348a3 in term 1.
I20260928 02:54:08.916236 3893 raft_consensus.cc:2454] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e7394af9a84c93b1d6124773b348a3 in term 1.
I20260928 02:54:08.916528 3697 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e7394af9a84c93b1d6124773b348a3, c79c4608e21a4ea58ad771a1b8c951ca; no voters:
I20260928 02:54:08.916703 3956 raft_consensus.cc:2781] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:08.916934 3956 raft_consensus.cc:686] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 1 LEADER]: Becoming Leader. State: Replica: 50e7394af9a84c93b1d6124773b348a3, State: Running, Role: LEADER
I20260928 02:54:08.917300 3956 consensus_queue.cc:237] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:08.920725 3480 catalog_manager.cc:5495] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130). New cstate: current_term: 1 leader_uuid: "50e7394af9a84c93b1d6124773b348a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } health_report { overall_health: UNKNOWN } } }
I20260928 02:54:08.950165 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.946869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.19.139.130:39707)
0928 02:54:08.946997 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:08.947000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:08.947013 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:54:08.947093 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:08.947095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:08.947150 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:08.947251 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:08.947453 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.947973 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.948868 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:08.949092 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:08.949169 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:08.949421 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:08.949851 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:08.949939 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:08.949991 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:54:08.950045 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:54:08.956632 3623 raft_consensus.cc:1261] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Refusing update from remote peer 50e7394af9a84c93b1d6124773b348a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:08.956632 3893 raft_consensus.cc:1261] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Refusing update from remote peer 50e7394af9a84c93b1d6124773b348a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:08.957363 3960 consensus_queue.cc:1035] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:08.957564 3956 consensus_queue.cc:1035] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:08.962674 3964 mvcc.cc:205] Tried to move back new op lower bound from 7334150344518283264 to 7334150344368414720. Current Snapshot: MvccSnapshot[applied={T|T < 7334150344518283264}]
I20260928 02:54:08.972296 3481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:08.972364 3968 mvcc.cc:205] Tried to move back new op lower bound from 7334150344518283264 to 7334150344368414720. Current Snapshot: MvccSnapshot[applied={T|T < 7334150344518283264}]
I20260928 02:54:08.972476 3481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:08.974957 3481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Sending DeleteTablet for 3 replicas of tablet 55e342e37efe4d939f3eee09605ec684
I20260928 02:54:08.975704 3603 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e342e37efe4d939f3eee09605ec684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:56018
I20260928 02:54:08.975826 3873 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e342e37efe4d939f3eee09605ec684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:42324
I20260928 02:54:08.975836 3872 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e342e37efe4d939f3eee09605ec684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:42324
I20260928 02:54:08.975975 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3519
I20260928 02:54:08.976787 3741 tablet_service.cc:1514] Processing DeleteTablet for tablet 55e342e37efe4d939f3eee09605ec684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:08 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:49832
I20260928 02:54:08.977257 3975 tablet_replica.cc:330] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: stopping tablet replica
I20260928 02:54:08.977521 3975 raft_consensus.cc:2229] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:54:08.977810 3975 raft_consensus.cc:2258] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:08.979107 3975 ts_tablet_manager.cc:1905] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:08.981743 3975 ts_tablet_manager.cc:1918] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 02:54:08.981788 3468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:39269 (error 108)
I20260928 02:54:08.981845 3975 log.cc:1196] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/55e342e37efe4d939f3eee09605ec684
I20260928 02:54:08.981876 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3673
I20260928 02:54:08.982174 3975 ts_tablet_manager.cc:1939] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Deleting consensus metadata
I20260928 02:54:08.982510 3974 tablet_replica.cc:330] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: stopping tablet replica
I20260928 02:54:08.982828 3974 raft_consensus.cc:2229] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:08.983194 3974 raft_consensus.cc:2258] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:08.983387 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.983019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:39269 (local address 127.0.0.1:56022)
0928 02:54:08.983217 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.983295 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 02:54:08.983544 3468 catalog_manager.cc:4584] TS c79c4608e21a4ea58ad771a1b8c951ca (127.19.139.129:39269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55e342e37efe4d939f3eee09605ec684: Network error: Client connection negotiation failed: client connection to 127.19.139.129:39269: connect: Connection refused (error 111)
I20260928 02:54:08.985175 3974 ts_tablet_manager.cc:1905] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:08.987323 3974 ts_tablet_manager.cc:1918] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:08.987425 3974 log.cc:1196] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/55e342e37efe4d939f3eee09605ec684
I20260928 02:54:08.987752 3974 ts_tablet_manager.cc:1939] T 55e342e37efe4d939f3eee09605ec684 P 88463cb9de6e4785bd7dba43d3fe767a: Deleting consensus metadata
I20260928 02:54:08.988039 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3810
I20260928 02:54:08.988245 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:08.987955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39707 (local address 127.0.0.1:49844)
0928 02:54:08.988078 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:08.988132 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:39707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:54:08.992987 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 3448
I20260928 02:54:28.998615 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37411
--webserver_interface=127.19.139.190
--webserver_port=34651
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:29.111843 3977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:29.112146 3977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:29.112242 3977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:29.116523 3977 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:54:29.116591 3977 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:54:29.116612 3977 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:54:29.116629 3977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:29.116654 3977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:29.116684 3977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:29.116714 3977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:29.116741 3977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:29.122339 3977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:29.123479 3977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:29.124595 3977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:29.133481 3983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:29.133548 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.133592 3977 server_base.cc:1034] running on GCE node
W20260928 02:54:29.133543 3982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.134402 3977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:29.135579 3977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:29.136747 3977 hybrid_clock.cc:648] HybridClock initialized: now 1790564069136714 us; error 50 us; skew 500 ppm
Sep 28 02:54:29 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:29.142984 3977 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:54:29.143996 3977 webserver.cc:460] Webserver started at http://127.19.139.190:34651/ using document root <none> and password file <none>
I20260928 02:54:29.144611 3977 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:29.144685 3977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:29.148702 3977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:29.151060 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.152477 3977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:29.152607 3977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "cda06403ce1e41bfa66dc70a25c26d05"
format_stamp: "Formatted at 2026-09-28 02:53:46 on dist-test-slave-5vc0"
server_key: "8171132a6791ae481bafbd0629aa3413"
server_key_iv: "0fa154c74e7d1137b38671af0386122f"
server_key_version: "encryptionkey@0"
I20260928 02:54:29.153174 3977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:29.174506 3977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:29.177875 3977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:29.178067 3977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:29.185192 3977 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:37411
I20260928 02:54:29.185539 4044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:37411 every 8 connection(s)
I20260928 02:54:29.186302 3977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:29.190789 4045 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:54:29.192551 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap starting.
I20260928 02:54:29.195168 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 3977
I20260928 02:54:29.195683 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:39269
--local_ip_for_outbound_sockets=127.19.139.129
--tserver_master_addrs=127.19.139.190:37411
--webserver_port=45461
--webserver_interface=127.19.139.129
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:54:29.210455 4045 log.cc:824] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:29.222249 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:54:29.222642 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Bootstrap complete.
I20260928 02:54:29.226528 4045 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:29.226935 4045 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda06403ce1e41bfa66dc70a25c26d05, State: Initialized, Role: FOLLOWER
I20260928 02:54:29.227412 4045 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:29.227530 4045 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:29.227591 4045 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:29.227685 4045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:54:29.229645 4045 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:29.230033 4045 leader_election.cc:302] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda06403ce1e41bfa66dc70a25c26d05; no voters:
I20260928 02:54:29.230370 4045 leader_election.cc:288] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:54:29.230470 4049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:54:29.230677 4049 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [term 3 LEADER]: Becoming Leader. State: Replica: cda06403ce1e41bfa66dc70a25c26d05, State: Running, Role: LEADER
I20260928 02:54:29.231053 4049 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } }
I20260928 02:54:29.231709 4045 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:29.232549 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cda06403ce1e41bfa66dc70a25c26d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } } }
I20260928 02:54:29.232558 4050 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda06403ce1e41bfa66dc70a25c26d05. Latest consensus state: current_term: 3 leader_uuid: "cda06403ce1e41bfa66dc70a25c26d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda06403ce1e41bfa66dc70a25c26d05" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37411 } } }
I20260928 02:54:29.232689 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:29.232718 4050 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:29.233242 4058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:29.237914 4058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:29.238456 4058 catalog_manager.cc:1259] Loaded cluster ID: 28861521ec64406abc26885d1a87a43a
I20260928 02:54:29.238509 4058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:29.240372 4058 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:29.240964 4058 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Loaded TSK: 0
I20260928 02:54:29.241031 4058 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cda06403ce1e41bfa66dc70a25c26d05: Loaded TSK: 1
F20260928 02:54:29.241657 4058 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f60ea4009ad google::LogMessage::Fail() at ??:0
@ 0x7f60ea40388a google::LogMessage::SendToLog() at ??:0
@ 0x7f60ea4004de google::LogMessage::Flush() at ??:0
@ 0x7f60ea401a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f60ed0c418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f60ed0bcf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f60f4d657bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f60f4d3f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f60f4d40178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f60f4d81df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f60f4da082b std::function<>::operator()() at ??:0
@ 0x7f60f4d3fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f60f4d40f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f60f4d3d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f60f4d81640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f60ec6523d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60ec652c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f60ec6543f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f60ec643512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60edc876db start_thread at ??:0
@ 0x7f60e8efe61f clone at ??:0
W20260928 02:54:29.324544 4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:29.324819 4047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:29.324864 4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:29.329169 4047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:29.329228 4047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:29.329293 4047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:29.335021 4047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:39269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=45461
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:29.336087 4047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:29.337364 4047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:29.347324 4047 server_base.cc:1034] running on GCE node
W20260928 02:54:29.347344 4093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:29.347509 4092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:29.347324 4095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.348029 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:29.348930 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:29.350142 4047 hybrid_clock.cc:648] HybridClock initialized: now 1790564069350099 us; error 54 us; skew 500 ppm
Sep 28 02:54:29 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:29.353430 4047 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:54:29.354347 4047 webserver.cc:460] Webserver started at http://127.19.139.129:45461/ using document root <none> and password file <none>
I20260928 02:54:29.354908 4047 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:29.354969 4047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:29.359097 4047 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:54:29.361377 4102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.362892 4047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:29.363018 4047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "c79c4608e21a4ea58ad771a1b8c951ca"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "873be534afce8e65474373aa91affdbc"
server_key_iv: "f729f60acf1c14704fe994d1ced88dfc"
server_key_version: "encryptionkey@0"
I20260928 02:54:29.363588 4047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:29.401086 4047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:29.404716 4047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:29.404925 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:29.405561 4047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:29.406746 4109 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:54:29.412379 4047 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:54:29.412473 4047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 02:54:29.412545 4047 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:54:29.414542 4047 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:54:29.414595 4047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:54:29.414925 4109 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap starting.
I20260928 02:54:29.423979 4047 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:39269
I20260928 02:54:29.425014 4216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:39269 every 8 connection(s)
I20260928 02:54:29.425184 4047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:29.432938 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4047
I20260928 02:54:29.433368 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:39707
--local_ip_for_outbound_sockets=127.19.139.130
--tserver_master_addrs=127.19.139.190:37411
--webserver_port=45769
--webserver_interface=127.19.139.130
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:54:29.438267 4109 log.cc:824] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:29.445603 4109 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:54:29.445979 4109 tablet_bootstrap.cc:492] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Bootstrap complete.
I20260928 02:54:29.446931 4109 ts_tablet_manager.cc:1397] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.007s
I20260928 02:54:29.449497 4109 raft_consensus.cc:348] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:29.449957 4109 raft_consensus.cc:729] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79c4608e21a4ea58ad771a1b8c951ca, State: Initialized, Role: FOLLOWER
I20260928 02:54:29.450484 4109 consensus_queue.cc:260] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:29.451448 4109 ts_tablet_manager.cc:1428] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260928 02:54:29.484983 4219 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:54:29.427120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:36361)
0928 02:54:29.427599 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.427653 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:29.428518 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:29.484207 (+ 55689us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":126,"threads_started":1}
W20260928 02:54:29.485497 4217 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:37411 (0 consecutive failures): Network error: Failed to ping master at 127.19.139.190:37411: Client connection negotiation failed: client connection to 127.19.139.190:37411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:54:29.486234 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.485962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:34841)
0928 02:54:29.486087 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.486160 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:54:29.486893 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.486683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:43151)
0928 02:54:29.486787 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.486840 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:54:29.487047 4217 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:54:29.561960 4220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:29.562296 4220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:29.562363 4220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:29.566686 4220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:29.566788 4220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:29.566877 4220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:29.572515 4220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:39707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=45769
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:29.573638 4220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:29.574680 4220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:29.584534 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.584671 4220 server_base.cc:1034] running on GCE node
W20260928 02:54:29.584537 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:29.584571 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.585383 4220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:29.586122 4220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:29.587301 4220 hybrid_clock.cc:648] HybridClock initialized: now 1790564069587281 us; error 32 us; skew 500 ppm
Sep 28 02:54:29 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:29.590119 4220 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:54:29.590991 4220 webserver.cc:460] Webserver started at http://127.19.139.130:45769/ using document root <none> and password file <none>
I20260928 02:54:29.591540 4220 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:29.591607 4220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:29.595487 4220 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:54:29.597667 4237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.598924 4220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:29.599041 4220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "50e7394af9a84c93b1d6124773b348a3"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "ab2d1fb7e59f4825ae70d3379a59a1b3"
server_key_iv: "4da4f6d8b90cdca1c93aaf3449a87a9f"
server_key_version: "encryptionkey@0"
I20260928 02:54:29.599573 4220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:29.615809 4220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:29.619102 4220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:29.619390 4220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:29.620059 4220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:29.621235 4244 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 02:54:29.625072 4244 ts_tablet_manager.cc:1852] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 02:54:29.625469 4244 ts_tablet_manager.cc:1905] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:29.629631 4244 ts_tablet_manager.cc:1918] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:29.629858 4244 ts_tablet_manager.cc:1939] T 55e342e37efe4d939f3eee09605ec684 P 50e7394af9a84c93b1d6124773b348a3: Deleting consensus metadata
I20260928 02:54:29.630337 4220 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 02:54:29.630396 4220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260928 02:54:29.630466 4220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:29.630491 4220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.639279 4220 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:39707
I20260928 02:54:29.639350 4351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:39707 every 8 connection(s)
I20260928 02:54:29.640193 4220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:29.641561 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4220
I20260928 02:54:29.641961 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:46309
--local_ip_for_outbound_sockets=127.19.139.131
--tserver_master_addrs=127.19.139.190:37411
--webserver_port=39923
--webserver_interface=127.19.139.131
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:54:29.643843 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.642417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:60339)
0928 02:54:29.642920 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.643061 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
W20260928 02:54:29.644291 4352 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:37411 (0 consecutive failures): Network error: Failed to ping master at 127.19.139.190:37411: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
I20260928 02:54:29.644869 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.644651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:55261)
0928 02:54:29.644765 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.644811 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:54:29.645424 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.645252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:57173)
0928 02:54:29.645337 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.645376 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:54:29.645569 4352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:54:29.753593 4355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:29.753890 4355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:29.753949 4355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:29.758162 4355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:29.758245 4355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:29.758332 4355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:29.763918 4355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:46309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=39923
--tserver_master_addrs=127.19.139.190:37411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:29.765075 4355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:29.766111 4355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:29.775672 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:29.775765 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.775754 4355 server_base.cc:1034] running on GCE node
W20260928 02:54:29.775749 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:29.776458 4355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:29.777114 4355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:29.778290 4355 hybrid_clock.cc:648] HybridClock initialized: now 1790564069778258 us; error 44 us; skew 500 ppm
Sep 28 02:54:29 dist-test-slave-5vc0 krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:29.781066 4355 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:54:29.781955 4355 webserver.cc:460] Webserver started at http://127.19.139.131:39923/ using document root <none> and password file <none>
I20260928 02:54:29.782478 4355 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:29.782536 4355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:29.786324 4355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:29.788555 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.790027 4355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:29.790129 4355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "88463cb9de6e4785bd7dba43d3fe767a"
format_stamp: "Formatted at 2026-09-28 02:53:47 on dist-test-slave-5vc0"
server_key: "4b08e67341baf99527b8d5d0983b69e9"
server_key_iv: "fdc40d7623a479e85e21d176e5d3a9b8"
server_key_version: "encryptionkey@0"
I20260928 02:54:29.790700 4355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:29.803408 4355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:29.806622 4355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:29.806780 4355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:29.807337 4355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:29.808126 4355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:29.808215 4355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.808276 4355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:29.808298 4355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:29.816800 4355 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46309
I20260928 02:54:29.816828 4483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46309 every 8 connection(s)
I20260928 02:54:29.817828 4355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:29.818485 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4355
I20260928 02:54:29.821916 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.820112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:43459)
0928 02:54:29.820613 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.820719 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":126,"threads_started":1}
W20260928 02:54:29.822327 4484 heartbeater.cc:644] Failed to heartbeat to 127.19.139.190:37411 (0 consecutive failures): Network error: Failed to ping master at 127.19.139.190:37411: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
I20260928 02:54:29.822916 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.822683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:44769)
0928 02:54:29.822809 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.822859 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:54:29.823567 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:29.823355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:35735)
0928 02:54:29.823473 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:29.823516 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:54:29.823776 4484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:54:30.488252 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:30.487753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:54149)
0928 02:54:30.488047 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:30.488111 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":81,"threads_started":1}
I20260928 02:54:30.646482 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:30.646126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:53205)
0928 02:54:30.646334 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:30.646386 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260928 02:54:30.824867 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:30.824386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:48621)
0928 02:54:30.824653 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:30.824712 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260928 02:54:31.089488 4491 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:31.089692 4491 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:31.090799 4491 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
Sep 28 02:54:31 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.131@KRBTEST.COM
Sep 28 02:54:31 dist-test-slave-5vc0 krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564069, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.130@KRBTEST.COM
I20260928 02:54:31.105120 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.091105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46309 (local address 127.19.139.129:46969)
0928 02:54:31.091471 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:31.091487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:31.091611 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:31.093284 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:31.093291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:31.093867 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:31.094369 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:31.094384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.096208 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.096213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:31.097222 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:31.097239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.098061 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.098065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:31.098266 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:31.098921 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:31.098933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:31.100570 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:31.102706 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:31.102712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:31.102727 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:31.102999 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:31.103339 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:31.103344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:31.103480 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:31.104146 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:31.104176 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:31.104400 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 02:54:31.104562 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 02:54:31.104927 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":165,"spinlock_wait_cycles":22528,"thread_start_us":80,"threads_started":1}
I20260928 02:54:31.105131 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.091239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:46969 (local address 127.19.139.131:46309)
0928 02:54:31.091848 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:31.091870 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:31.091903 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:54:31.092563 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:31.092569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:31.092911 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:31.093193 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:31.094541 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.096034 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.097342 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.097947 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.098099 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:31.100803 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:31.100815 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:31.100829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:31.100860 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:31.102575 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103115 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:31.103120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:31.103124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:31.103242 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103568 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:31.103570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:31.103572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:31.104039 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:31.104180 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:31.104684 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 02:54:31.104948 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":129,"threads_started":1}
I20260928 02:54:31.105157 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.091105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39707 (local address 127.19.139.129:43817)
0928 02:54:31.091534 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:31.091548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:31.091667 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:31.093285 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:31.093291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:31.093867 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:31.094368 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:31.094384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.096200 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.096203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:31.097228 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:31.097239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.098061 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.098064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:31.098265 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:31.098915 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:31.098936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:31.100914 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:31.103011 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:31.103017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:31.103202 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:31.103513 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:31.103518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:31.103595 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:31.104302 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:31.104305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:31.104401 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:54:31.104562 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:54:31.104999 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":315,"spinlock_wait_cycles":9088,"thread_start_us":65,"threads_started":1}
I20260928 02:54:31.105139 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.091402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:43817 (local address 127.19.139.130:39707)
0928 02:54:31.091736 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:31.091752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:31.091794 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:54:31.092564 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:31.092571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:31.092911 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:31.093193 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:31.094548 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.096033 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.097365 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:31.097941 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:31.098099 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:31.101133 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:31.101139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:31.101151 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:31.101187 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:31.102916 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103303 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:31.103309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:31.103315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:31.103426 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:31.103671 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:31.103675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:31.103676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:31.104178 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:31.104313 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:31.104681 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:54:31.104953 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
W20260928 02:54:31.107597 4104 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
W20260928 02:54:31.107758 4103 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
I20260928 02:54:31.107851 4103 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c79c4608e21a4ea58ad771a1b8c951ca; no voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a
I20260928 02:54:31.108109 4491 raft_consensus.cc:2726] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:54:31.489310 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.489036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:32795)
0928 02:54:31.489169 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:31.489227 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:54:31.647722 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.647271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:40033)
0928 02:54:31.647564 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:31.647627 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20260928 02:54:31.826109 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:31.825613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:55221)
0928 02:54:31.825944 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:31.826013 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":75,"threads_started":1}
I20260928 02:54:32.490490 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:32.490080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:34943)
0928 02:54:32.490332 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:32.490396 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260928 02:54:32.648993 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:32.648579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:56935)
0928 02:54:32.648829 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:32.648890 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260928 02:54:32.827291 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:32.826881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:47021)
0928 02:54:32.827135 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:32.827195 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260928 02:54:32.995652 4501 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:32.995806 4501 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:32.996299 4501 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
W20260928 02:54:32.997138 4103 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
W20260928 02:54:32.997300 4104 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
I20260928 02:54:32.997411 4104 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c79c4608e21a4ea58ad771a1b8c951ca; no voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a
I20260928 02:54:32.997560 4501 raft_consensus.cc:2726] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:54:33.491755 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:33.491292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:46713)
0928 02:54:33.491579 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:33.491640 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260928 02:54:33.650214 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:33.649688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:49661)
0928 02:54:33.649999 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:33.650073 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260928 02:54:33.828624 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:33.828111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:39139)
0928 02:54:33.828437 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:33.828518 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":131,"threads_started":1}
I20260928 02:54:34.493253 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:34.492766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:50133)
0928 02:54:34.493086 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:34.493155 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":75,"threads_started":1}
I20260928 02:54:34.651494 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:34.651040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:44363)
0928 02:54:34.651325 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:34.651392 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260928 02:54:34.829906 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:34.829496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:39079)
0928 02:54:34.829741 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:34.829805 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260928 02:54:34.876098 4508 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:34.876267 4508 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:34.876708 4508 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
W20260928 02:54:34.877411 4103 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
W20260928 02:54:34.877516 4104 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
I20260928 02:54:34.877579 4104 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c79c4608e21a4ea58ad771a1b8c951ca; no voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a
I20260928 02:54:34.877698 4508 raft_consensus.cc:2726] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:54:35.494693 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:35.494169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:40635)
0928 02:54:35.494520 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:35.494591 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260928 02:54:35.652776 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:35.652342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:59961)
0928 02:54:35.652617 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:35.652676 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260928 02:54:35.831064 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:35.830643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:50135)
0928 02:54:35.830910 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:35.830972 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260928 02:54:36.495981 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:36.495478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:56525)
0928 02:54:36.495816 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:36.495884 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
I20260928 02:54:36.653915 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:36.653466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:39465)
0928 02:54:36.653763 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:36.653818 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260928 02:54:36.832363 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:36.831839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:35849)
0928 02:54:36.832195 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:36.832265 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":90,"threads_started":1}
I20260928 02:54:37.497110 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:37.496685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:34927)
0928 02:54:37.496957 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:37.497015 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260928 02:54:37.655205 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:37.654740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:47033)
0928 02:54:37.655039 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:37.655106 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260928 02:54:37.734999 4517 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:37.735136 4517 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:37.735512 4517 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
W20260928 02:54:37.736333 4103 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
W20260928 02:54:37.736490 4104 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
I20260928 02:54:37.736591 4104 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c79c4608e21a4ea58ad771a1b8c951ca; no voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a
I20260928 02:54:37.736815 4517 raft_consensus.cc:2726] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:54:37.833688 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:37.833242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:55271)
0928 02:54:37.833522 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:37.833588 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260928 02:54:38.498225 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:38.497807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:36755)
0928 02:54:38.498078 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:38.498132 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260928 02:54:38.656481 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:38.656004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:60713)
0928 02:54:38.656295 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:38.656365 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260928 02:54:38.835119 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:38.834594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:59093)
0928 02:54:38.834948 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:38.835018 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":100,"threads_started":1}
I20260928 02:54:39.499643 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:39.499187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:42835)
0928 02:54:39.499472 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:39.499537 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260928 02:54:39.657725 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:39.657263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:46419)
0928 02:54:39.657544 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:39.657615 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260928 02:54:39.836616 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:39.836092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:59591)
0928 02:54:39.836439 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:39.836506 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
I20260928 02:54:40.501063 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:40.500534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:44597)
0928 02:54:40.500884 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:40.500955 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":91,"threads_started":1}
I20260928 02:54:40.659024 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:40.658578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:50129)
0928 02:54:40.658863 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:40.658921 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
I20260928 02:54:40.837940 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:40.837487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:40631)
0928 02:54:40.837773 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:40.837839 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260928 02:54:41.502445 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:41.501952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:39423)
0928 02:54:41.502276 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:41.502344 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
I20260928 02:54:41.660424 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:41.659877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:33495)
0928 02:54:41.660228 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:41.660302 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260928 02:54:41.839139 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:41.838725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:60041)
0928 02:54:41.838977 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:41.839041 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":137,"threads_started":1}
I20260928 02:54:42.400249 4531 raft_consensus.cc:482] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:42.400389 4531 raft_consensus.cc:504] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88463cb9de6e4785bd7dba43d3fe767a" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46309 } } peers { permanent_uuid: "50e7394af9a84c93b1d6124773b348a3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39707 } } peers { permanent_uuid: "c79c4608e21a4ea58ad771a1b8c951ca" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 39269 } }
I20260928 02:54:42.400828 4531 leader_election.cc:288] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309), 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707)
W20260928 02:54:42.401535 4104 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88463cb9de6e4785bd7dba43d3fe767a (127.19.139.131:46309): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
W20260928 02:54:42.401631 4103 leader_election.cc:341] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50e7394af9a84c93b1d6124773b348a3 (127.19.139.130:39707): Not found: Tablet not found: 55e342e37efe4d939f3eee09605ec684
I20260928 02:54:42.401713 4103 leader_election.cc:302] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c79c4608e21a4ea58ad771a1b8c951ca; no voters: 50e7394af9a84c93b1d6124773b348a3, 88463cb9de6e4785bd7dba43d3fe767a
I20260928 02:54:42.401893 4531 raft_consensus.cc:2726] T 55e342e37efe4d939f3eee09605ec684 P c79c4608e21a4ea58ad771a1b8c951ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:54:42.503593 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:42.503205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:39867)
0928 02:54:42.503444 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:42.503499 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260928 02:54:42.661770 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:42.661308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:33581)
0928 02:54:42.661564 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:42.661653 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260928 02:54:42.840478 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:42.839975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:41731)
0928 02:54:42.840310 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:42.840378 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":94,"threads_started":1}
I20260928 02:54:43.504920 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:43.504350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:58727)
0928 02:54:43.504696 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:43.504784 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
I20260928 02:54:43.662930 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:43.662460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:53561)
0928 02:54:43.662749 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:43.662815 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":98,"threads_started":1}
I20260928 02:54:43.841709 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:43.841274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.131:35469)
0928 02:54:43.841545 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:43.841611 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260928 02:54:44.506428 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:44.506001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.129:44993)
0928 02:54:44.506263 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:44.506329 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260928 02:54:44.664247 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:44.663782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37411 (local address 127.19.139.130:41465)
0928 02:54:44.664049 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:44.664109 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.190:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260928 02:54:44.828680 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4047
I20260928 02:54:44.833961 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4220
I20260928 02:54:44.838680 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4355
2026-09-28T02:54:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59101 ms)
[----------] 30 tests from SecurityITest (239150 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:44 dist-test-slave-5vc0 krb5kdc[4542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:44 dist-test-slave-5vc0 krb5kdc[4542](info): set up 2 sockets
Sep 28 02:54:44 dist-test-slave-5vc0 krb5kdc[4542](info): commencing operation
krb5kdc: starting...
W20260928 02:54:45.799901 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:54:45 dist-test-slave-5vc0 krb5kdc[4542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:54:45.921663 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35651
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:35651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:46.032487 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:46.032814 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:46.032859 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:46.037021 4555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:46.037089 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:46.037107 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:46.037123 4555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:46.037137 4555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:46.042595 4555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:35651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:46.043735 4555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:46.044844 4555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:46.055042 4555 server_base.cc:1034] running on GCE node
W20260928 02:54:46.054976 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.055081 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.054852 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:46.055955 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:46.057225 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:46.058423 4555 hybrid_clock.cc:648] HybridClock initialized: now 1790564086058403 us; error 37 us; skew 500 ppm
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:46.061487 4555 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:54:46.062476 4555 webserver.cc:460] Webserver started at http://127.19.139.190:46043/ using document root <none> and password file <none>
I20260928 02:54:46.063031 4555 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:46.063097 4555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:46.063333 4555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:46.065690 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "baca8b7bbab84a3a9967ed5578dbcf11"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "e0c16f4869e5d1eeb9a59aca04406b3c"
server_key_iv: "f454c802c9c137c7b3c4bdc6cec8e4ce"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.066344 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "baca8b7bbab84a3a9967ed5578dbcf11"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "e0c16f4869e5d1eeb9a59aca04406b3c"
server_key_iv: "f454c802c9c137c7b3c4bdc6cec8e4ce"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.070292 4555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:54:46.073046 4570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.074471 4555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:46.074611 4555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "baca8b7bbab84a3a9967ed5578dbcf11"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "e0c16f4869e5d1eeb9a59aca04406b3c"
server_key_iv: "f454c802c9c137c7b3c4bdc6cec8e4ce"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.074733 4555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:46.100178 4555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:46.103715 4555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:46.103914 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:46.110772 4555 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:35651
I20260928 02:54:46.110819 4622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:35651 every 8 connection(s)
I20260928 02:54:46.111779 4555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:46.114727 4623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:46.118381 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4555
I20260928 02:54:46.118507 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:46.118827 20014 external_mini_cluster.cc:1428] Setting key caeb456243cffbc4938fb0e02e6a4116
I20260928 02:54:46.120977 4623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: Bootstrap starting.
I20260928 02:54:46.123489 4623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:46.124336 4623 log.cc:824] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:46.126659 4623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: No bootstrap required, opened a new log
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:46.130306 4623 raft_consensus.cc:348] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35651 } }
I20260928 02:54:46.130555 4623 raft_consensus.cc:374] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:46.130591 4623 raft_consensus.cc:729] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baca8b7bbab84a3a9967ed5578dbcf11, State: Initialized, Role: FOLLOWER
I20260928 02:54:46.131063 4623 consensus_queue.cc:260] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35651 } }
I20260928 02:54:46.131248 4623 raft_consensus.cc:388] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:46.131322 4623 raft_consensus.cc:482] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:46.131428 4623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:46.132536 4623 raft_consensus.cc:504] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35651 } }
I20260928 02:54:46.132944 4623 leader_election.cc:302] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baca8b7bbab84a3a9967ed5578dbcf11; no voters:
I20260928 02:54:46.133353 4623 leader_election.cc:288] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:46.133452 4628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:46.133617 4628 raft_consensus.cc:686] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [term 1 LEADER]: Becoming Leader. State: Replica: baca8b7bbab84a3a9967ed5578dbcf11, State: Running, Role: LEADER
I20260928 02:54:46.134004 4628 consensus_queue.cc:237] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35651 } }
I20260928 02:54:46.134128 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.120261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.19.139.190:35651)
0928 02:54:46.120694 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.120710 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.120746 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:54:46.121442 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.121462 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.121812 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.122136 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:46.122976 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.124048 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.125153 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.125645 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.125767 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.128619 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:46.128625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:46.128640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:46.128672 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:46.130965 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.131472 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.131477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.131483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.131629 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.131922 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.131924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.131925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.132421 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:46.132583 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.132894 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.133201 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
I20260928 02:54:46.134712 4623 sys_catalog.cc:564] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:46.135248 4628 sys_catalog.cc:455] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baca8b7bbab84a3a9967ed5578dbcf11. Latest consensus state: current_term: 1 leader_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35651 } } }
I20260928 02:54:46.135963 4628 sys_catalog.cc:458] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:46.135849 4629 sys_catalog.cc:455] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baca8b7bbab84a3a9967ed5578dbcf11" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35651 } } }
I20260928 02:54:46.136309 4629 sys_catalog.cc:458] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11 [sys.catalog]: This master's current role is: LEADER
W20260928 02:54:46.139586 4643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:54:46.139657 4643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:54:46.139732 4637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:46.140471 4637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:46.145960 4637 catalog_manager.cc:1347] Generated new cluster ID: 60cdc815473047aaae6c2c2fe9a99d56
I20260928 02:54:46.146049 4637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:46.153549 4637 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:46.154893 4637 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:46.160773 4637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: Generated new TSK 0
I20260928 02:54:46.161687 4637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:54:46.191495 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35651
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:46.305683 4649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:46.305995 4649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:46.306056 4649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:46.310148 4649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:46.310220 4649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:46.310304 4649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:46.315732 4649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:46.316943 4649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:46.317997 4649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:46.327543 4654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:46.327721 4649 server_base.cc:1034] running on GCE node
W20260928 02:54:46.327647 4655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.327543 4657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:46.328305 4649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:46.329042 4649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:46.330209 4649 hybrid_clock.cc:648] HybridClock initialized: now 1790564086330179 us; error 45 us; skew 500 ppm
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:46.333148 4649 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:54:46.334213 4649 webserver.cc:460] Webserver started at http://127.19.139.129:46371/ using document root <none> and password file <none>
I20260928 02:54:46.334811 4649 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:46.334877 4649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:46.335052 4649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:46.337294 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "cddc0d4500fc4892b2a9e672cbb00de1"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "4ca50a84dfa5b8a0674eb399957ab62d"
server_key_iv: "e88c8ad9374d7c6470e2b9092a25f20f"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.337877 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "cddc0d4500fc4892b2a9e672cbb00de1"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "4ca50a84dfa5b8a0674eb399957ab62d"
server_key_iv: "e88c8ad9374d7c6470e2b9092a25f20f"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.342024 4649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:54:46.344898 4664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.346477 4649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:46.346625 4649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "cddc0d4500fc4892b2a9e672cbb00de1"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "4ca50a84dfa5b8a0674eb399957ab62d"
server_key_iv: "e88c8ad9374d7c6470e2b9092a25f20f"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.346762 4649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:46.367846 4649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:46.371280 4649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:46.371488 4649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:46.372116 4649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:46.373029 4649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:46.373080 4649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.373132 4649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:46.373162 4649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.382203 4649 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:43851
I20260928 02:54:46.382229 4777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:43851 every 8 connection(s)
I20260928 02:54:46.383199 4649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:46.388685 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4649
I20260928 02:54:46.388809 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:46.389087 20014 external_mini_cluster.cc:1428] Setting key 668f20aef58f928a4d6499b3bf509c07
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:54:46.397832 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.385087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:49565 (local address 127.19.139.190:35651)
0928 02:54:46.385233 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.385237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.386205 (+ 968us) server_negotiation.cc:407] Connection header received
0928 02:54:46.387598 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.387602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.387664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.387743 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:46.389102 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.389817 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.391180 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.391427 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.391515 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.394153 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:46.394157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:46.394159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:46.394189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:46.395618 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.396198 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.396202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.396203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.396301 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.396667 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.396671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.396673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.396885 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:46.396974 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.397516 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.397659 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:54:46.398761 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.385494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35651 (local address 127.19.139.129:49565)
0928 02:54:46.386026 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:46.386075 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:46.387362 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:46.387954 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:46.387962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:46.388393 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:46.388954 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.388971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.389972 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.389976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.391058 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.391069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.391587 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.391591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.391770 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.392417 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:46.392435 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:46.393949 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:46.395754 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:46.395761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:46.395782 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:46.396072 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:46.396391 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:46.396394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:46.396396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:46.396573 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:46.396987 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:46.396995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:46.397282 (+ 287us) client_negotiation.cc:769] Sending connection context
0928 02:54:46.397462 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 02:54:46.397845 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
I20260928 02:54:46.400434 4778 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35651
I20260928 02:54:46.400724 4778 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:46.401419 4778 heartbeater.cc:511] Master 127.19.139.190:35651 requested a full tablet report, sending...
I20260928 02:54:46.403451 4587 ts_manager.cc:194] Registered new tserver with Master: cddc0d4500fc4892b2a9e672cbb00de1 (127.19.139.129:43851)
I20260928 02:54:46.404826 4587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:49565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:54:46.420430 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35651
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:46.532641 4783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:46.532960 4783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:46.533023 4783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:46.537312 4783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:46.537391 4783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:46.537473 4783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:46.543061 4783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:46.544188 4783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:46.545241 4783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:46.555095 4783 server_base.cc:1034] running on GCE node
W20260928 02:54:46.555054 4788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.555050 4791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.555050 4789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:46.555709 4783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:46.556463 4783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:46.557629 4783 hybrid_clock.cc:648] HybridClock initialized: now 1790564086557613 us; error 32 us; skew 500 ppm
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:46.560516 4783 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:54:46.561753 4783 webserver.cc:460] Webserver started at http://127.19.139.130:36005/ using document root <none> and password file <none>
I20260928 02:54:46.562322 4783 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:46.562393 4783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:46.562592 4783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:46.564815 4783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "fd3ac5e454cf4bc4ab22bb522646500b"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "8e74dcc376dd66d3e964c79d0d151abc"
server_key_iv: "cbe20d2751c4c60482def6018a18fea2"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.565433 4783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "fd3ac5e454cf4bc4ab22bb522646500b"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "8e74dcc376dd66d3e964c79d0d151abc"
server_key_iv: "cbe20d2751c4c60482def6018a18fea2"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.569342 4783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:54:46.571856 4798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.573279 4783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:46.573390 4783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "fd3ac5e454cf4bc4ab22bb522646500b"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "8e74dcc376dd66d3e964c79d0d151abc"
server_key_iv: "cbe20d2751c4c60482def6018a18fea2"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.573504 4783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:46.586848 4783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:46.590131 4783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:46.590317 4783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:46.590915 4783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:46.591782 4783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:46.591830 4783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.591872 4783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:46.591900 4783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.600289 4783 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:39883
I20260928 02:54:46.600338 4911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:39883 every 8 connection(s)
I20260928 02:54:46.601289 4783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:46.606930 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4783
I20260928 02:54:46.607060 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:46.607336 20014 external_mini_cluster.cc:1428] Setting key a45ef6e95cf74cf9c34eedb7273f3096
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:54:46.616114 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.603302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:51161 (local address 127.19.139.190:35651)
0928 02:54:46.603455 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.603461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.604461 (+ 1000us) server_negotiation.cc:407] Connection header received
0928 02:54:46.605873 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.605877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.605929 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.606017 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:46.607342 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.608012 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.609368 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.609599 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.609701 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.612717 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:46.612720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:46.612722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:46.612748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:46.614076 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.614588 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.614591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.614592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.614672 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.615007 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.615009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.615011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.615203 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:46.615287 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.615807 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.615938 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:54:46.617007 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.603706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35651 (local address 127.19.139.130:51161)
0928 02:54:46.604301 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:46.604350 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:46.605635 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:46.606212 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:46.606220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:46.606593 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:46.607189 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.607206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.608146 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.608174 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.609226 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.609241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.609831 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.609835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.609986 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.610603 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:46.610628 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:46.612537 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:46.614191 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:46.614198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:46.614216 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:46.614483 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:46.614751 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:46.614754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:46.614756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:46.614912 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:46.615300 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:46.615307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:46.615567 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 02:54:46.615733 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 02:54:46.616098 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":128,"threads_started":1}
I20260928 02:54:46.618156 4912 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35651
I20260928 02:54:46.618399 4912 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:46.618932 4912 heartbeater.cc:511] Master 127.19.139.190:35651 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
I20260928 02:54:46.620007 4587 ts_manager.cc:194] Registered new tserver with Master: fd3ac5e454cf4bc4ab22bb522646500b (127.19.139.130:39883)
I20260928 02:54:46.620623 4587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:51161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:54:46.636339 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35651
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:46.748005 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:46.748351 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:46.748402 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:46.752785 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:46.752852 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:46.752913 4917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:46.758409 4917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:35651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:46.759470 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:46.760524 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:46.770274 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.770360 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:46.770274 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:46.771365 4917 server_base.cc:1034] running on GCE node
I20260928 02:54:46.771833 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:46.772547 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:46.773774 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790564086773742 us; error 46 us; skew 500 ppm
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:46.776717 4917 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:54:46.777726 4917 webserver.cc:460] Webserver started at http://127.19.139.131:42059/ using document root <none> and password file <none>
I20260928 02:54:46.778323 4917 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:46.778419 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:46.778638 4917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:46.780993 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "5902258f7ef943a2af488ceb771220ed"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "2ea7b280695cb4a12968bed18efa78b3"
server_key_iv: "6a2cea5cc06b3885f69e258f50b0dc64"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.781605 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "5902258f7ef943a2af488ceb771220ed"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "2ea7b280695cb4a12968bed18efa78b3"
server_key_iv: "6a2cea5cc06b3885f69e258f50b0dc64"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.786118 4917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:54:46.788764 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.790086 4917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:46.790205 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "5902258f7ef943a2af488ceb771220ed"
format_stamp: "Formatted at 2026-09-28 02:54:46 on dist-test-slave-5vc0"
server_key: "2ea7b280695cb4a12968bed18efa78b3"
server_key_iv: "6a2cea5cc06b3885f69e258f50b0dc64"
server_key_version: "encryptionkey@0"
I20260928 02:54:46.790316 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:46.811879 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:46.814999 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:46.815196 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:46.815796 4917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:46.816704 4917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:46.816752 4917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.816807 4917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:46.816835 4917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:46.825480 4917 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:41907
I20260928 02:54:46.825503 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:41907 every 8 connection(s)
I20260928 02:54:46.826579 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:46.833302 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 4917
I20260928 02:54:46.833461 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:46.833829 20014 external_mini_cluster.cc:1428] Setting key 048d98aa43769e8b034294fba4d05299
Sep 28 02:54:46 dist-test-slave-5vc0 krb5kdc[4542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564086, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:46.840981 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.828474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:53147 (local address 127.19.139.190:35651)
0928 02:54:46.828609 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.828613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.829551 (+ 938us) server_negotiation.cc:407] Connection header received
0928 02:54:46.830939 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.830942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.831002 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.831085 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:46.832382 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.832941 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.834564 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.834848 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.834940 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.837487 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:46.837491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:46.837494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:46.837523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:46.838841 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.839358 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.839362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.839364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.839445 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.839762 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.839765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.839767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.840012 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:46.840100 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.840673 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.840814 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:54:46.841857 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.828880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:35651 (local address 127.19.139.131:53147)
0928 02:54:46.829388 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:46.829433 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:46.830698 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:46.831279 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:46.831288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:46.831666 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:46.832211 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.832227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.833093 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.833097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.834416 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.834427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.835005 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.835009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.835200 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.835795 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:46.835812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:46.837324 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:46.838969 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:46.838975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:46.838994 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:46.839262 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:46.839528 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:46.839531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:46.839533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:46.839676 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:46.840113 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:46.840120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:46.840439 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 02:54:46.840615 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 02:54:46.840960 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260928 02:54:46.843029 5046 heartbeater.cc:347] Connected to a master server at 127.19.139.190:35651
I20260928 02:54:46.843257 5046 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:46.843750 5046 heartbeater.cc:511] Master 127.19.139.190:35651 requested a full tablet report, sending...
I20260928 02:54:46.844885 4587 ts_manager.cc:194] Registered new tserver with Master: 5902258f7ef943a2af488ceb771220ed (127.19.139.131:41907)
I20260928 02:54:46.845434 4587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:53147
I20260928 02:54:46.847761 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:46.856676 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.849682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.19.139.190:35651)
0928 02:54:46.849866 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.849871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.850002 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:54:46.850106 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.850109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.850150 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.850212 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:46.850875 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.851435 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.852346 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.852555 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.852638 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.853916 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:46.853919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:46.853921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:46.853944 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:46.855097 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.855454 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.855457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.855458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.855529 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:46.855734 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:46.855736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:46.855737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:46.855971 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:46.856073 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.856387 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.856508 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:54:46.859696 4587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:46.862996 4587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:46.878207 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.872124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:41907 (local address 127.0.0.1:41656)
0928 02:54:46.872908 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:46.872921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:46.873000 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:46.874249 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:46.874252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:46.874266 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:46.874379 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.874386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.875352 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.875356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.876727 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.876739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.877667 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.877671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.877791 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.877835 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:54:46.877939 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:54:46.878067 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260928 02:54:46.878213 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.872992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.19.139.131:41907)
0928 02:54:46.873611 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.873623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.873644 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:54:46.873718 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.873723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.873940 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.874101 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:46.874492 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.875219 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.876862 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.877552 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.877667 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.877912 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.877967 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.878057 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":141,"threads_started":1}
I20260928 02:54:46.878697 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.872081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43851 (local address 127.0.0.1:36246)
0928 02:54:46.872495 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:46.872548 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:46.872679 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:46.874127 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:46.874130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:46.874149 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:46.874286 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.874301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.875819 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.875823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.877039 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.877048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.878213 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.878217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.878342 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.878367 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:54:46.878426 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:54:46.878573 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":129,"thread_start_us":127,"threads_started":1}
I20260928 02:54:46.878939 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.872218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.19.139.129:43851)
0928 02:54:46.873140 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.873156 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.873657 (+ 501us) server_negotiation.cc:407] Connection header received
0928 02:54:46.873736 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.873740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.873983 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.874437 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:46.874580 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.875676 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.877162 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.878091 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.878526 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.878656 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.878698 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.878820 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":177,"threads_started":1}
I20260928 02:54:46.878927 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.873288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.19.139.130:39883)
0928 02:54:46.873591 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:46.873601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:46.873624 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:54:46.873706 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:46.873712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:46.873926 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:46.874075 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:46.874396 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.875667 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.877138 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.877999 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.878120 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.878577 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:46.878630 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:54:46.878776 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260928 02:54:46.879431 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:46.872971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39883 (local address 127.0.0.1:49840)
0928 02:54:46.873265 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:46.873277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:46.873362 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:46.874087 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:46.874093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:46.874117 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:46.874267 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.874273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.875869 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.875872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.877018 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:46.877027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:46.878277 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:46.878280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:46.878383 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:46.878401 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:54:46.878449 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:54:46.879267 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":694,"spinlock_wait_cycles":896,"thread_start_us":105,"threads_started":1}
I20260928 02:54:46.881742 4980 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca (DEFAULT_TABLE table=test-table [id=7b3fff888b794aeab421a1979420b471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:46.881894 4712 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca (DEFAULT_TABLE table=test-table [id=7b3fff888b794aeab421a1979420b471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:46.882720 4980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:46.882913 4712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:46.882556 4846 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca (DEFAULT_TABLE table=test-table [id=7b3fff888b794aeab421a1979420b471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:46.883494 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:46.889340 5063 tablet_bootstrap.cc:492] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Bootstrap starting.
I20260928 02:54:46.890180 5064 tablet_bootstrap.cc:492] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: Bootstrap starting.
I20260928 02:54:46.890883 5065 tablet_bootstrap.cc:492] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Bootstrap starting.
I20260928 02:54:46.891983 5063 tablet_bootstrap.cc:654] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:46.892282 5064 tablet_bootstrap.cc:654] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:46.892593 5065 tablet_bootstrap.cc:654] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:46.893400 5064 log.cc:824] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:46.893389 5063 log.cc:824] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:46.893617 5065 log.cc:824] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:46.895569 5064 tablet_bootstrap.cc:492] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: No bootstrap required, opened a new log
I20260928 02:54:46.895569 5063 tablet_bootstrap.cc:492] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: No bootstrap required, opened a new log
I20260928 02:54:46.895687 5065 tablet_bootstrap.cc:492] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: No bootstrap required, opened a new log
I20260928 02:54:46.895778 5064 ts_tablet_manager.cc:1397] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 02:54:46.895778 5063 ts_tablet_manager.cc:1397] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:54:46.895884 5065 ts_tablet_manager.cc:1397] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:54:46.898851 5065 raft_consensus.cc:348] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:46.899097 5065 raft_consensus.cc:374] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:46.899344 5065 raft_consensus.cc:729] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3ac5e454cf4bc4ab22bb522646500b, State: Initialized, Role: FOLLOWER
I20260928 02:54:46.899649 5063 raft_consensus.cc:348] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:46.899649 5064 raft_consensus.cc:348] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:46.899811 5065 consensus_queue.cc:260] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:46.899957 5064 raft_consensus.cc:374] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:46.899957 5063 raft_consensus.cc:374] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:46.900017 5064 raft_consensus.cc:729] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddc0d4500fc4892b2a9e672cbb00de1, State: Initialized, Role: FOLLOWER
I20260928 02:54:46.900017 5063 raft_consensus.cc:729] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5902258f7ef943a2af488ceb771220ed, State: Initialized, Role: FOLLOWER
I20260928 02:54:46.900496 4912 heartbeater.cc:503] Master 127.19.139.190:35651 was elected leader, sending a full tablet report...
I20260928 02:54:46.900492 5064 consensus_queue.cc:260] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:46.900488 5063 consensus_queue.cc:260] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:46.900874 5065 ts_tablet_manager.cc:1428] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:46.901398 5063 ts_tablet_manager.cc:1428] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:54:46.901407 5064 ts_tablet_manager.cc:1428] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:54:46.901717 4778 heartbeater.cc:503] Master 127.19.139.190:35651 was elected leader, sending a full tablet report...
I20260928 02:54:46.901717 5046 heartbeater.cc:503] Master 127.19.139.190:35651 was elected leader, sending a full tablet report...
I20260928 02:54:47.037141 5070 raft_consensus.cc:482] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:47.037365 5070 raft_consensus.cc:504] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:47.038489 5070 leader_election.cc:288] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5902258f7ef943a2af488ceb771220ed (127.19.139.131:41907), fd3ac5e454cf4bc4ab22bb522646500b (127.19.139.130:39883)
I20260928 02:54:47.042304 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.038901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:41569 (local address 127.19.139.130:39883)
0928 02:54:47.039103 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.039107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.039303 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:54:47.039388 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.039391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.039435 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.039496 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:47.039772 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.040294 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.041375 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.041762 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.041868 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.041924 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.042090 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.042159 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260928 02:54:47.042322 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.038801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39883 (local address 127.19.139.129:41569)
0928 02:54:47.039179 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.039191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.039298 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.039520 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.039523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.039550 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:47.039668 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.039677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.040397 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.040401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.041252 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.041262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.041882 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.041886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.041982 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.042000 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.042062 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.042136 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":3,"spinlock_wait_cycles":8320,"thread_start_us":87,"threads_started":1}
I20260928 02:54:47.042363 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.038766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:41907 (local address 127.19.139.129:54243)
0928 02:54:47.039133 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.039147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.039276 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.039521 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.039524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.039557 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:47.039695 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.039701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.040521 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.040524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.041335 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.041344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.041948 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.041951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.042046 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.042068 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.042133 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.042201 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":4,"spinlock_wait_cycles":11776,"thread_start_us":133,"threads_started":1}
I20260928 02:54:47.042341 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.038865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:54243 (local address 127.19.139.131:41907)
0928 02:54:47.038997 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.039001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.039240 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:54:47.039389 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.039392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.039435 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.039501 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:47.039801 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.040417 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.041455 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.041840 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.041945 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.041994 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.042157 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.042224 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:54:47.042980 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3e7d1cd75a433fb1ac21da3df8a1ca" candidate_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" is_pre_election: true
I20260928 02:54:47.042984 5000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3e7d1cd75a433fb1ac21da3df8a1ca" candidate_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5902258f7ef943a2af488ceb771220ed" is_pre_election: true
I20260928 02:54:47.043319 5000 raft_consensus.cc:2454] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddc0d4500fc4892b2a9e672cbb00de1 in term 0.
I20260928 02:54:47.043319 4866 raft_consensus.cc:2454] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddc0d4500fc4892b2a9e672cbb00de1 in term 0.
I20260928 02:54:47.043833 4667 leader_election.cc:302] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5902258f7ef943a2af488ceb771220ed, cddc0d4500fc4892b2a9e672cbb00de1, fd3ac5e454cf4bc4ab22bb522646500b; no voters:
I20260928 02:54:47.044101 5070 raft_consensus.cc:2781] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:47.044209 5070 raft_consensus.cc:482] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:47.044260 5070 raft_consensus.cc:3037] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:47.045269 5070 raft_consensus.cc:504] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:47.045594 5070 leader_election.cc:288] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [CANDIDATE]: Term 1 election: Requested vote from peers 5902258f7ef943a2af488ceb771220ed (127.19.139.131:41907), fd3ac5e454cf4bc4ab22bb522646500b (127.19.139.130:39883)
I20260928 02:54:47.045943 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3e7d1cd75a433fb1ac21da3df8a1ca" candidate_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3ac5e454cf4bc4ab22bb522646500b"
I20260928 02:54:47.045953 5000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3e7d1cd75a433fb1ac21da3df8a1ca" candidate_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5902258f7ef943a2af488ceb771220ed"
I20260928 02:54:47.046051 4866 raft_consensus.cc:3037] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:47.046078 5000 raft_consensus.cc:3037] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:47.047052 4866 raft_consensus.cc:2454] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddc0d4500fc4892b2a9e672cbb00de1 in term 1.
I20260928 02:54:47.047140 5000 raft_consensus.cc:2454] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddc0d4500fc4892b2a9e672cbb00de1 in term 1.
I20260928 02:54:47.047363 4666 leader_election.cc:302] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cddc0d4500fc4892b2a9e672cbb00de1, fd3ac5e454cf4bc4ab22bb522646500b; no voters:
I20260928 02:54:47.047531 5070 raft_consensus.cc:2781] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:47.047747 5070 raft_consensus.cc:686] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 1 LEADER]: Becoming Leader. State: Replica: cddc0d4500fc4892b2a9e672cbb00de1, State: Running, Role: LEADER
I20260928 02:54:47.048074 5070 consensus_queue.cc:237] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } }
I20260928 02:54:47.051440 4587 catalog_manager.cc:5495] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cddc0d4500fc4892b2a9e672cbb00de1 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddc0d4500fc4892b2a9e672cbb00de1" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 43851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 } health_report { overall_health: UNKNOWN } } }
I20260928 02:54:47.069664 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.066050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.19.139.129:43851)
0928 02:54:47.066159 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.066163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.066176 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:54:47.066322 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.066324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.066383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.066479 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:47.066781 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.067332 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.068332 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.068551 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.068641 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.068934 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:47.069362 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:47.069449 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.069500 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.069552 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:54:47.075762 5000 raft_consensus.cc:1261] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 1 FOLLOWER]: Refusing update from remote peer cddc0d4500fc4892b2a9e672cbb00de1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:47.075762 4866 raft_consensus.cc:1261] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 1 FOLLOWER]: Refusing update from remote peer cddc0d4500fc4892b2a9e672cbb00de1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:47.076534 5070 consensus_queue.cc:1035] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5902258f7ef943a2af488ceb771220ed" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:47.076742 5074 consensus_queue.cc:1035] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3ac5e454cf4bc4ab22bb522646500b" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:54:47.077485 5047 tablet.cc:2369] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:47.083590 5077 mvcc.cc:205] Tried to move back new op lower bound from 7334150500654628864 to 7334150500551823360. Current Snapshot: MvccSnapshot[applied={T|T < 7334150500654628864}]
I20260928 02:54:47.084540 5078 mvcc.cc:205] Tried to move back new op lower bound from 7334150500654628864 to 7334150500551823360. Current Snapshot: MvccSnapshot[applied={T|T < 7334150500654628864}]
I20260928 02:54:47.085721 5079 mvcc.cc:205] Tried to move back new op lower bound from 7334150500654628864 to 7334150500551823360. Current Snapshot: MvccSnapshot[applied={T|T < 7334150500654628864}]
I20260928 02:54:47.093954 4587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:47.094125 4587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:47.096509 4587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baca8b7bbab84a3a9967ed5578dbcf11: Sending DeleteTablet for 3 replicas of tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca
I20260928 02:54:47.097317 4846 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:49840
I20260928 02:54:47.097290 4980 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:41656
I20260928 02:54:47.097342 4712 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:47 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:36246
I20260928 02:54:47.097863 5086 tablet_replica.cc:330] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: stopping tablet replica
I20260928 02:54:47.098093 5088 tablet_replica.cc:330] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1: stopping tablet replica
I20260928 02:54:47.098430 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4649
I20260928 02:54:47.098466 5088 raft_consensus.cc:2229] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P cddc0d4500fc4892b2a9e672cbb00de1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:54:47.099134 5086 raft_consensus.cc:2229] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:47.099439 5086 raft_consensus.cc:2258] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:47.099798 5087 tablet_replica.cc:330] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: stopping tablet replica
I20260928 02:54:47.100073 5087 raft_consensus.cc:2229] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:47.100387 5087 raft_consensus.cc:2258] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:47.101727 5087 ts_tablet_manager.cc:1905] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:47.101727 5086 ts_tablet_manager.cc:1905] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:54:47.104429 4572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:43851 (error 108)
I20260928 02:54:47.104497 5086 ts_tablet_manager.cc:1918] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:47.104497 5087 ts_tablet_manager.cc:1918] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:47.104595 5086 log.cc:1196] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/7b3e7d1cd75a433fb1ac21da3df8a1ca
I20260928 02:54:47.104595 5087 log.cc:1196] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/7b3e7d1cd75a433fb1ac21da3df8a1ca
I20260928 02:54:47.104897 5087 ts_tablet_manager.cc:1939] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P 5902258f7ef943a2af488ceb771220ed: Deleting consensus metadata
I20260928 02:54:47.104895 5086 ts_tablet_manager.cc:1939] T 7b3e7d1cd75a433fb1ac21da3df8a1ca P fd3ac5e454cf4bc4ab22bb522646500b: Deleting consensus metadata
I20260928 02:54:47.105532 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.105193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:43851 (local address 127.0.0.1:36272)
0928 02:54:47.105365 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.105443 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:43851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:54:47.105526 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4783
W20260928 02:54:47.106199 4572 catalog_manager.cc:4584] TS cddc0d4500fc4892b2a9e672cbb00de1 (127.19.139.129:43851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca: Network error: Client connection negotiation failed: client connection to 127.19.139.129:43851: connect: Connection refused (error 111)
I20260928 02:54:47.106608 4573 catalog_manager.cc:4841] TS 5902258f7ef943a2af488ceb771220ed (127.19.139.131:41907): tablet 7b3e7d1cd75a433fb1ac21da3df8a1ca (table test-table [id=7b3fff888b794aeab421a1979420b471]) successfully deleted
I20260928 02:54:47.110867 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4917
I20260928 02:54:47.112255 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.111009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:39883 (local address 127.0.0.1:49844)
0928 02:54:47.112096 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.112178 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":997}
I20260928 02:54:47.115720 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 4555
2026-09-28T02:54:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2276 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:47Z Disabled control of system clock
I20260928 02:54:47.144148 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45087
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:54:47.256393 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:47.256690 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:47.256726 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:47.260808 5093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:47.260874 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:47.260888 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:47.260905 5093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:47.260918 5093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:47.265798 5093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:47.266916 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:47.268324 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:47.276799 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:47.276805 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:47.276968 5093 server_base.cc:1034] running on GCE node
W20260928 02:54:47.276823 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:47.277684 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:47.278820 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:47.279978 5093 hybrid_clock.cc:648] HybridClock initialized: now 1790564087279949 us; error 42 us; skew 500 ppm
I20260928 02:54:47.281895 5093 webserver.cc:460] Webserver started at http://127.19.139.190:33985/ using document root <none> and password file <none>
I20260928 02:54:47.282467 5093 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:47.282532 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:47.282730 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:47.284825 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "8539df6e4f7e464db398783147af3e1b"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "2c6573e05b815a245383e67bfe1086ff"
server_key_iv: "fba985f5c8bbd2a5fe307bb192e19120"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.285372 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "8539df6e4f7e464db398783147af3e1b"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "2c6573e05b815a245383e67bfe1086ff"
server_key_iv: "fba985f5c8bbd2a5fe307bb192e19120"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.288955 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:47.291339 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.292627 5093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:47.292747 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "8539df6e4f7e464db398783147af3e1b"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "2c6573e05b815a245383e67bfe1086ff"
server_key_iv: "fba985f5c8bbd2a5fe307bb192e19120"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.292850 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:47.306674 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:47.307404 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:47.307585 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:47.314640 5093 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:45087
I20260928 02:54:47.314648 5159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:45087 every 8 connection(s)
I20260928 02:54:47.315620 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:47.318753 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:47.320032 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5093
I20260928 02:54:47.320196 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:47.320506 20014 external_mini_cluster.cc:1428] Setting key 064f59ca71ab700e79a9cc51d43aacd5
I20260928 02:54:47.324867 5160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b: Bootstrap starting.
I20260928 02:54:47.327039 5160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:47.327862 5160 log.cc:824] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:47.329643 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.321930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.19.139.190:45087)
0928 02:54:47.322576 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.322590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.322627 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:54:47.323305 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.323323 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.323718 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.324034 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:47.324487 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.325467 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.326326 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.326792 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.326938 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.327792 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:47.327797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:47.327810 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:47.328272 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:47.328343 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:47.328371 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:47.328472 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.328630 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.328919 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20260928 02:54:47.329911 5160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b: No bootstrap required, opened a new log
I20260928 02:54:47.333606 5160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8539df6e4f7e464db398783147af3e1b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45087 } }
I20260928 02:54:47.333902 5160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:47.333963 5160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8539df6e4f7e464db398783147af3e1b, State: Initialized, Role: FOLLOWER
I20260928 02:54:47.334412 5160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8539df6e4f7e464db398783147af3e1b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45087 } }
I20260928 02:54:47.334542 5160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:47.334606 5160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:47.334712 5160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:47.335632 5160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8539df6e4f7e464db398783147af3e1b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45087 } }
I20260928 02:54:47.335979 5160 leader_election.cc:302] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8539df6e4f7e464db398783147af3e1b; no voters:
I20260928 02:54:47.336311 5160 leader_election.cc:288] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:47.336431 5165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:47.336663 5165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [term 1 LEADER]: Becoming Leader. State: Replica: 8539df6e4f7e464db398783147af3e1b, State: Running, Role: LEADER
I20260928 02:54:47.337040 5165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8539df6e4f7e464db398783147af3e1b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45087 } }
I20260928 02:54:47.337545 5160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:47.338665 5167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8539df6e4f7e464db398783147af3e1b. Latest consensus state: current_term: 1 leader_uuid: "8539df6e4f7e464db398783147af3e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8539df6e4f7e464db398783147af3e1b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45087 } } }
I20260928 02:54:47.338792 5167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:47.339038 5166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8539df6e4f7e464db398783147af3e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8539df6e4f7e464db398783147af3e1b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45087 } } }
I20260928 02:54:47.339130 5166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:47.339226 5174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:47.342368 5174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:47.347460 5174 catalog_manager.cc:1347] Generated new cluster ID: 2a613a78da354292a42e8a73c25c4005
I20260928 02:54:47.347534 5174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:47.373809 5174 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:47.374748 5174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:47.380442 5174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b: Generated new TSK 0
I20260928 02:54:47.381102 5174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:54:47.385144 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45087
--builtin_ntp_servers=127.19.139.148:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:54:47.496533 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:47.496892 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:47.496964 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:47.501101 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:47.501158 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:47.501240 5184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:47.506078 5184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:47.507162 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:47.508586 5184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:47.518002 5190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:47.518067 5189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:47.518061 5184 server_base.cc:1034] running on GCE node
W20260928 02:54:47.518213 5192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:47.518589 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:47.519235 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:47.520394 5184 hybrid_clock.cc:648] HybridClock initialized: now 1790564087520374 us; error 34 us; skew 500 ppm
I20260928 02:54:47.522250 5184 webserver.cc:460] Webserver started at http://127.19.139.129:42733/ using document root <none> and password file <none>
I20260928 02:54:47.522815 5184 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:47.522881 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:47.523062 5184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:47.525187 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "7c6df6c63cda48d18d248a401b84838a"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "5e47e2a5770d841435c0bb586100d062"
server_key_iv: "4290a16fee0652c330536aa7413cce17"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.525722 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "7c6df6c63cda48d18d248a401b84838a"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "5e47e2a5770d841435c0bb586100d062"
server_key_iv: "4290a16fee0652c330536aa7413cce17"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.529397 5184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:47.531777 5198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.533172 5184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:47.533284 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "7c6df6c63cda48d18d248a401b84838a"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "5e47e2a5770d841435c0bb586100d062"
server_key_iv: "4290a16fee0652c330536aa7413cce17"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.533407 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:47.551429 5184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:47.552227 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:47.552407 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:47.553014 5184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:47.553977 5184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:47.554024 5184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.554077 5184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:47.554105 5184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.562695 5184 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:36089
I20260928 02:54:47.562733 5311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:36089 every 8 connection(s)
I20260928 02:54:47.563608 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:47.571632 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5184
I20260928 02:54:47.571766 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:47.572068 20014 external_mini_cluster.cc:1428] Setting key 746dc88f5d27ae3e1fea91724b2afa48
I20260928 02:54:47.572932 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.565483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:56627 (local address 127.19.139.190:45087)
0928 02:54:47.565607 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.565610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.566647 (+ 1037us) server_negotiation.cc:407] Connection header received
0928 02:54:47.568055 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.568058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.568122 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.568246 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:47.569156 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.569704 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.570854 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.571087 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.571174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.572422 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:47.572426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:47.572429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:47.572443 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:47.572467 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:47.572476 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:47.572547 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.572723 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.572791 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:54:47.574025 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.565816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45087 (local address 127.19.139.129:56627)
0928 02:54:47.566493 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.566533 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.567840 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.568412 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.568420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.568510 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:47.569015 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.569029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.569827 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.569830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.570739 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.570748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.571242 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.571247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.571402 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.572116 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:47.572118 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:47.572186 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:47.572269 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:47.572567 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:47.572574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:47.572579 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.572718 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.572974 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260928 02:54:47.574149 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45087
--builtin_ntp_servers=127.19.139.148:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:54:47.575299 5312 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45087
I20260928 02:54:47.575523 5312 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:47.576067 5312 heartbeater.cc:511] Master 127.19.139.190:45087 requested a full tablet report, sending...
I20260928 02:54:47.577706 5124 ts_manager.cc:194] Registered new tserver with Master: 7c6df6c63cda48d18d248a401b84838a (127.19.139.129:36089)
I20260928 02:54:47.579344 5124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.129:56627
W20260928 02:54:47.682636 5315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:47.682909 5315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:47.682945 5315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:47.686921 5315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:47.686971 5315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:47.687022 5315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:47.691620 5315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:47.692595 5315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:47.693905 5315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:47.703226 5323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:47.703233 5321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:47.703281 5320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:47.703591 5315 server_base.cc:1034] running on GCE node
I20260928 02:54:47.704011 5315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:47.704687 5315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:47.705869 5315 hybrid_clock.cc:648] HybridClock initialized: now 1790564087705836 us; error 48 us; skew 500 ppm
I20260928 02:54:47.707651 5315 webserver.cc:460] Webserver started at http://127.19.139.130:34921/ using document root <none> and password file <none>
I20260928 02:54:47.708244 5315 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:47.708317 5315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:47.708518 5315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:47.710601 5315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "516aeb10bae54ed0a0b67fc72a01f4ea"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "99599b2aaec0e3efa32efe44fd5cfa88"
server_key_iv: "01c0027467f8a112824e62fcb3a1d508"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.711159 5315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "516aeb10bae54ed0a0b67fc72a01f4ea"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "99599b2aaec0e3efa32efe44fd5cfa88"
server_key_iv: "01c0027467f8a112824e62fcb3a1d508"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.714820 5315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:54:47.717360 5329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.718622 5315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:47.718724 5315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "516aeb10bae54ed0a0b67fc72a01f4ea"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "99599b2aaec0e3efa32efe44fd5cfa88"
server_key_iv: "01c0027467f8a112824e62fcb3a1d508"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.718837 5315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:47.749948 5315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:47.750645 5315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:47.750808 5315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:47.751400 5315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:47.752357 5315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:47.752403 5315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.752449 5315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:47.752476 5315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.760730 5315 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:33661
I20260928 02:54:47.760782 5442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:33661 every 8 connection(s)
I20260928 02:54:47.761663 5315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:47.770804 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.763537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:44103 (local address 127.19.139.190:45087)
0928 02:54:47.763706 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.763709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.764777 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 02:54:47.766154 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.766158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.766217 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.766289 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:47.767187 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.767708 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.768891 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.769118 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.769209 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.770336 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:47.770339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:47.770341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:47.770353 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:47.770372 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:47.770379 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:47.770439 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.770589 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.770653 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:54:47.771287 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5315
I20260928 02:54:47.771411 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:47.771667 20014 external_mini_cluster.cc:1428] Setting key b373b10084eac9c58904d46ed776d0a2
I20260928 02:54:47.771742 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.763934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45087 (local address 127.19.139.130:44103)
0928 02:54:47.764629 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.764672 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.765953 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.766474 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.766481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.766568 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:47.767053 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.767066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.767823 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.767826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.768777 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.768787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.769260 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.769265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.769429 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.770089 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:47.770091 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:47.770130 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:47.770196 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:47.770457 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:47.770467 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:47.770471 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.770582 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.770867 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260928 02:54:47.772948 5443 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45087
I20260928 02:54:47.773191 5443 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:47.773700 5443 heartbeater.cc:511] Master 127.19.139.190:45087 requested a full tablet report, sending...
I20260928 02:54:47.773706 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45087
--builtin_ntp_servers=127.19.139.148:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:54:47.774689 5123 ts_manager.cc:194] Registered new tserver with Master: 516aeb10bae54ed0a0b67fc72a01f4ea (127.19.139.130:33661)
I20260928 02:54:47.775234 5123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.130:44103
W20260928 02:54:47.885916 5446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:47.886206 5446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:47.886245 5446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:47.890352 5446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:47.890417 5446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:47.890482 5446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:47.895401 5446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:47.896595 5446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:47.897998 5446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:47.907640 5446 server_base.cc:1034] running on GCE node
W20260928 02:54:47.907598 5452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:47.907598 5451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:47.907600 5454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:47.908269 5446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:47.908929 5446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:47.910118 5446 hybrid_clock.cc:648] HybridClock initialized: now 1790564087910090 us; error 41 us; skew 500 ppm
I20260928 02:54:47.912067 5446 webserver.cc:460] Webserver started at http://127.19.139.131:38421/ using document root <none> and password file <none>
I20260928 02:54:47.912694 5446 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:47.912763 5446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:47.912955 5446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:47.915156 5446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "00839f70b26e453c9326babdfabbf9ca"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "2abce3618b3a313ffad54f4b342126f2"
server_key_iv: "17928d198665e432d1045291655a2054"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.915709 5446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "00839f70b26e453c9326babdfabbf9ca"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "2abce3618b3a313ffad54f4b342126f2"
server_key_iv: "17928d198665e432d1045291655a2054"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.919411 5446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 02:54:47.921885 5460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.923126 5446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:47.923246 5446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "00839f70b26e453c9326babdfabbf9ca"
format_stamp: "Formatted at 2026-09-28 02:54:47 on dist-test-slave-5vc0"
server_key: "2abce3618b3a313ffad54f4b342126f2"
server_key_iv: "17928d198665e432d1045291655a2054"
server_key_version: "encryptionkey@0"
I20260928 02:54:47.923367 5446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:47.942858 5446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:47.943596 5446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:47.943770 5446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:47.944388 5446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:47.945340 5446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:47.945402 5446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.945461 5446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:47.945485 5446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:47.953711 5446 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:33021
I20260928 02:54:47.953742 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:33021 every 8 connection(s)
I20260928 02:54:47.954599 5446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:47.960187 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5446
I20260928 02:54:47.960314 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:47.960566 20014 external_mini_cluster.cc:1428] Setting key 0096c94ba1101b15d0ff65611e0b0cd8
I20260928 02:54:47.963960 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.956359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:44263 (local address 127.19.139.190:45087)
0928 02:54:47.956501 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.956504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.957633 (+ 1129us) server_negotiation.cc:407] Connection header received
0928 02:54:47.959052 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.959056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.959118 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.959193 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:47.960178 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.960792 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.962163 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.962400 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.962491 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.963541 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:47.963544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:47.963546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:47.963558 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:47.963578 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:47.963585 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:47.963641 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.963752 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.963805 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:54:47.964902 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.956790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45087 (local address 127.19.139.131:44263)
0928 02:54:47.957469 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.957511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.958808 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.959395 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.959402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.959497 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:47.960002 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.960015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.960916 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.960919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.962052 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.962062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.962553 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.962556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.962746 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.963288 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:47.963290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:47.963331 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:47.963405 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:47.963647 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:47.963651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:47.963655 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.963744 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.964028 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260928 02:54:47.965934 5574 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45087
I20260928 02:54:47.966152 5574 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:47.966656 5574 heartbeater.cc:511] Master 127.19.139.190:45087 requested a full tablet report, sending...
I20260928 02:54:47.967617 5123 ts_manager.cc:194] Registered new tserver with Master: 00839f70b26e453c9326babdfabbf9ca (127.19.139.131:33021)
I20260928 02:54:47.968235 5123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.131:44263
I20260928 02:54:47.974399 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:47.979785 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.976381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.19.139.190:45087)
0928 02:54:47.976504 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.976507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.976655 (+ 148us) server_negotiation.cc:407] Connection header received
0928 02:54:47.976764 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.976766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.976806 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.976901 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:47.977094 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.977594 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.978436 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.978643 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.978712 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.979468 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:47.979471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:47.979473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:47.979485 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:47.979501 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:47.979509 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:47.979563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.979624 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.979675 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:54:47.982473 5123 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:47.985617 5123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:48.000008 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.994003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.19.139.129:36089)
0928 02:54:47.994498 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.994511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.994536 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:54:47.994616 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.994621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.994845 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.995010 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:47.995445 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.996716 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.998241 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.998968 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.999067 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.999254 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:47.999717 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 02:54:47.999827 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":127,"threads_started":1}
I20260928 02:54:48.000025 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.993856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:36089 (local address 127.0.0.1:43066)
0928 02:54:47.994243 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.994301 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.994437 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.995140 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.995146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.995167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:47.995309 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.995317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.996831 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.996835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.998099 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.998109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.999362 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.999366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.999564 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.999586 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.999688 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.999837 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":67,"thread_start_us":102,"threads_started":1}
I20260928 02:54:48.000029 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.994630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:33021 (local address 127.0.0.1:46840)
0928 02:54:47.995257 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.995269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.995366 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.996088 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.996090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.996109 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:47.996222 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.996229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.997490 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.997493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.998580 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.998588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.999336 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.999339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.999499 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.999563 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.999675 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.999855 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":88,"thread_start_us":103,"threads_started":1}
I20260928 02:54:48.000030 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.994036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33661 (local address 127.0.0.1:53514)
0928 02:54:47.994940 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:47.994953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:47.995026 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:47.995782 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:47.995785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:47.995803 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:47.995910 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.995916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.997241 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.997244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.998463 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:47.998474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.999601 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.999605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:47.999687 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.999705 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:54:47.999764 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:54:47.999846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":520,"thread_start_us":87,"threads_started":1}
I20260928 02:54:48.000562 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.995154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.19.139.131:33021)
0928 02:54:47.995615 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.995625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.995647 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:54:47.995717 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.995723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.995908 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.996092 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:47.996354 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.997365 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.998702 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.999221 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.999332 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.999524 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:48.000294 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 02:54:48.000409 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":149,"threads_started":1}
I20260928 02:54:48.001379 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:47.994280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.19.139.130:33661)
0928 02:54:47.995375 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:47.995398 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:47.995423 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:54:47.995500 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:47.995504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:47.995649 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:47.995786 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:47.996013 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.997126 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.998591 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:47.999341 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:47.999448 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:47.999615 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:48.001102 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0928 02:54:48.001215 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":141,"threads_started":1}
I20260928 02:54:48.003218 5246 tablet_service.cc:1467] Processing CreateTablet for tablet b79a68cb1d924e30880a8f030e157b33 (DEFAULT_TABLE table=test-table [id=4a91953a6ac34dae84f582e52d00db07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:48.003342 5508 tablet_service.cc:1467] Processing CreateTablet for tablet b79a68cb1d924e30880a8f030e157b33 (DEFAULT_TABLE table=test-table [id=4a91953a6ac34dae84f582e52d00db07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:48.003796 5377 tablet_service.cc:1467] Processing CreateTablet for tablet b79a68cb1d924e30880a8f030e157b33 (DEFAULT_TABLE table=test-table [id=4a91953a6ac34dae84f582e52d00db07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:48.004245 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79a68cb1d924e30880a8f030e157b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:48.004443 5508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79a68cb1d924e30880a8f030e157b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:48.004758 5377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79a68cb1d924e30880a8f030e157b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:48.010994 5591 tablet_bootstrap.cc:492] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: Bootstrap starting.
I20260928 02:54:48.012400 5592 tablet_bootstrap.cc:492] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Bootstrap starting.
I20260928 02:54:48.012457 5593 tablet_bootstrap.cc:492] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Bootstrap starting.
I20260928 02:54:48.013550 5591 tablet_bootstrap.cc:654] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:48.014282 5591 log.cc:824] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:48.014616 5592 tablet_bootstrap.cc:654] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:48.014616 5593 tablet_bootstrap.cc:654] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:48.015455 5593 log.cc:824] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:48.015455 5592 log.cc:824] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:48.016909 5591 tablet_bootstrap.cc:492] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: No bootstrap required, opened a new log
I20260928 02:54:48.017122 5591 ts_tablet_manager.cc:1397] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:54:48.017582 5592 tablet_bootstrap.cc:492] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: No bootstrap required, opened a new log
I20260928 02:54:48.017771 5592 ts_tablet_manager.cc:1397] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:54:48.017856 5593 tablet_bootstrap.cc:492] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: No bootstrap required, opened a new log
I20260928 02:54:48.018049 5593 ts_tablet_manager.cc:1397] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:54:48.020972 5592 raft_consensus.cc:348] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.021214 5592 raft_consensus.cc:374] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:48.021275 5592 raft_consensus.cc:729] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00839f70b26e453c9326babdfabbf9ca, State: Initialized, Role: FOLLOWER
I20260928 02:54:48.021466 5591 raft_consensus.cc:348] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.021744 5591 raft_consensus.cc:374] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:48.021683 5592 consensus_queue.cc:260] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.021800 5591 raft_consensus.cc:729] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6df6c63cda48d18d248a401b84838a, State: Initialized, Role: FOLLOWER
I20260928 02:54:48.022364 5574 heartbeater.cc:503] Master 127.19.139.190:45087 was elected leader, sending a full tablet report...
I20260928 02:54:48.022312 5591 consensus_queue.cc:260] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.022332 5593 raft_consensus.cc:348] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.022557 5593 raft_consensus.cc:374] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:48.022605 5593 raft_consensus.cc:729] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516aeb10bae54ed0a0b67fc72a01f4ea, State: Initialized, Role: FOLLOWER
I20260928 02:54:48.022758 5592 ts_tablet_manager.cc:1428] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:54:48.023054 5593 consensus_queue.cc:260] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.023397 5591 ts_tablet_manager.cc:1428] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:54:48.023699 5443 heartbeater.cc:503] Master 127.19.139.190:45087 was elected leader, sending a full tablet report...
I20260928 02:54:48.023950 5593 ts_tablet_manager.cc:1428] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:54:48.024809 5312 heartbeater.cc:503] Master 127.19.139.190:45087 was elected leader, sending a full tablet report...
W20260928 02:54:48.064919 5313 tablet.cc:2369] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:54:48.205559 5575 tablet.cc:2369] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:54:48.262802 5444 tablet.cc:2369] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:48.385185 5597 raft_consensus.cc:482] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:48.385401 5597 raft_consensus.cc:504] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.386552 5597 leader_election.cc:288] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6df6c63cda48d18d248a401b84838a (127.19.139.129:36089), 516aeb10bae54ed0a0b67fc72a01f4ea (127.19.139.130:33661)
I20260928 02:54:48.390360 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:48.386846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:36089 (local address 127.19.139.131:52347)
0928 02:54:48.386961 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:48.386977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:48.387074 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:48.387490 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:48.387493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:48.387517 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:48.387640 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:48.387648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.388496 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.388500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:48.389338 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:48.389347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.389948 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.389952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:48.390059 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:48.390076 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:54:48.390133 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:54:48.390199 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":3}
I20260928 02:54:48.390372 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:48.387014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:44841 (local address 127.19.139.130:33661)
0928 02:54:48.387149 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:48.387154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:48.387443 (+ 289us) server_negotiation.cc:407] Connection header received
0928 02:54:48.387554 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:48.387557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:48.387605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:48.387704 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:48.387966 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.388494 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.389467 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.389845 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.389948 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:48.389997 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:48.390206 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:54:48.390274 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:54:48.390367 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:48.387092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:52347 (local address 127.19.139.129:36089)
0928 02:54:48.387253 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:48.387258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:48.387273 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:54:48.387326 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:48.387329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:48.387380 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:48.387463 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:48.387805 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.388387 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.389468 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.389841 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.389945 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:48.389997 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:48.390168 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 02:54:48.390240 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:54:48.390384 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:48.386846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33661 (local address 127.19.139.131:44841)
0928 02:54:48.387351 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:48.387363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:48.387462 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:48.387716 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:48.387719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:48.387735 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:48.387859 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:48.387866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.388595 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.388598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:48.389344 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:48.389352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.389964 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.389968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:48.390090 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:48.390107 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:54:48.390179 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:54:48.390247 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":96,"thread_start_us":113,"threads_started":1}
I20260928 02:54:48.391004 5266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79a68cb1d924e30880a8f030e157b33" candidate_uuid: "00839f70b26e453c9326babdfabbf9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6df6c63cda48d18d248a401b84838a" is_pre_election: true
I20260928 02:54:48.391041 5397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79a68cb1d924e30880a8f030e157b33" candidate_uuid: "00839f70b26e453c9326babdfabbf9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" is_pre_election: true
I20260928 02:54:48.391286 5397 raft_consensus.cc:2454] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00839f70b26e453c9326babdfabbf9ca in term 0.
I20260928 02:54:48.391295 5266 raft_consensus.cc:2454] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00839f70b26e453c9326babdfabbf9ca in term 0.
I20260928 02:54:48.391718 5462 leader_election.cc:302] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00839f70b26e453c9326babdfabbf9ca, 516aeb10bae54ed0a0b67fc72a01f4ea; no voters:
I20260928 02:54:48.391970 5597 raft_consensus.cc:2781] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:48.392050 5597 raft_consensus.cc:482] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:48.392097 5597 raft_consensus.cc:3037] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:48.393096 5597 raft_consensus.cc:504] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.393409 5597 leader_election.cc:288] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [CANDIDATE]: Term 1 election: Requested vote from peers 7c6df6c63cda48d18d248a401b84838a (127.19.139.129:36089), 516aeb10bae54ed0a0b67fc72a01f4ea (127.19.139.130:33661)
I20260928 02:54:48.393802 5266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79a68cb1d924e30880a8f030e157b33" candidate_uuid: "00839f70b26e453c9326babdfabbf9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6df6c63cda48d18d248a401b84838a"
I20260928 02:54:48.393802 5397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79a68cb1d924e30880a8f030e157b33" candidate_uuid: "00839f70b26e453c9326babdfabbf9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea"
I20260928 02:54:48.393947 5266 raft_consensus.cc:3037] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:48.393947 5397 raft_consensus.cc:3037] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:48.395303 5266 raft_consensus.cc:2454] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00839f70b26e453c9326babdfabbf9ca in term 1.
I20260928 02:54:48.395303 5397 raft_consensus.cc:2454] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00839f70b26e453c9326babdfabbf9ca in term 1.
I20260928 02:54:48.395632 5461 leader_election.cc:302] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00839f70b26e453c9326babdfabbf9ca, 7c6df6c63cda48d18d248a401b84838a; no voters:
I20260928 02:54:48.395834 5597 raft_consensus.cc:2781] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:48.396056 5597 raft_consensus.cc:686] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 1 LEADER]: Becoming Leader. State: Replica: 00839f70b26e453c9326babdfabbf9ca, State: Running, Role: LEADER
I20260928 02:54:48.396443 5597 consensus_queue.cc:237] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } }
I20260928 02:54:48.399575 5123 catalog_manager.cc:5495] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 00839f70b26e453c9326babdfabbf9ca (127.19.139.131). New cstate: current_term: 1 leader_uuid: "00839f70b26e453c9326babdfabbf9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00839f70b26e453c9326babdfabbf9ca" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 } health_report { overall_health: UNKNOWN } } }
I20260928 02:54:48.453648 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:48.450101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.19.139.131:33021)
0928 02:54:48.450235 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:48.450239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:48.450250 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:54:48.450338 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:48.450340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:48.450396 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:48.450481 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:48.450767 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.451280 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.452175 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:48.452393 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:48.452465 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:48.452836 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:48.453261 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:48.453364 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:48.453471 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:54:48.453522 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:54:48.459563 5266 raft_consensus.cc:1261] T b79a68cb1d924e30880a8f030e157b33 P 7c6df6c63cda48d18d248a401b84838a [term 1 FOLLOWER]: Refusing update from remote peer 00839f70b26e453c9326babdfabbf9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:48.459563 5397 raft_consensus.cc:1261] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 1 FOLLOWER]: Refusing update from remote peer 00839f70b26e453c9326babdfabbf9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:48.460330 5597 consensus_queue.cc:1035] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "516aeb10bae54ed0a0b67fc72a01f4ea" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:48.460577 5601 consensus_queue.cc:1035] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6df6c63cda48d18d248a401b84838a" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 36089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:48.468142 5604 mvcc.cc:205] Tried to move back new op lower bound from 7334150506322944000 to 7334150506074636288. Current Snapshot: MvccSnapshot[applied={T|T < 7334150506322944000}]
I20260928 02:54:48.476599 5123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:48.476812 5123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:48.479336 5123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8539df6e4f7e464db398783147af3e1b: Sending DeleteTablet for 3 replicas of tablet b79a68cb1d924e30880a8f030e157b33
I20260928 02:54:48.480229 5377 tablet_service.cc:1514] Processing DeleteTablet for tablet b79a68cb1d924e30880a8f030e157b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:48 UTC) from {username='slave'} at 127.0.0.1:53514
I20260928 02:54:48.480249 5508 tablet_service.cc:1514] Processing DeleteTablet for tablet b79a68cb1d924e30880a8f030e157b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:48 UTC) from {username='slave'} at 127.0.0.1:46840
I20260928 02:54:48.480340 5246 tablet_service.cc:1514] Processing DeleteTablet for tablet b79a68cb1d924e30880a8f030e157b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:48 UTC) from {username='slave'} at 127.0.0.1:43066
I20260928 02:54:48.480774 5613 tablet_replica.cc:330] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: stopping tablet replica
I20260928 02:54:48.480860 5614 tablet_replica.cc:330] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: stopping tablet replica
I20260928 02:54:48.481173 5614 raft_consensus.cc:2229] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:54:48.481173 5613 raft_consensus.cc:2229] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:48.481431 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5184
I20260928 02:54:48.482805 5613 raft_consensus.cc:2258] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:48.484548 5614 raft_consensus.cc:2258] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:48.484563 5613 ts_tablet_manager.cc:1905] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:48.486081 5614 ts_tablet_manager.cc:1905] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:48.487414 5613 ts_tablet_manager.cc:1918] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:48.487521 5613 log.cc:1196] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/b79a68cb1d924e30880a8f030e157b33
I20260928 02:54:48.487825 5613 ts_tablet_manager.cc:1939] T b79a68cb1d924e30880a8f030e157b33 P 516aeb10bae54ed0a0b67fc72a01f4ea: Deleting consensus metadata
I20260928 02:54:48.488904 5614 ts_tablet_manager.cc:1918] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:48.489058 5614 log.cc:1196] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/b79a68cb1d924e30880a8f030e157b33
I20260928 02:54:48.489233 5109 catalog_manager.cc:4841] TS 516aeb10bae54ed0a0b67fc72a01f4ea (127.19.139.130:33661): tablet b79a68cb1d924e30880a8f030e157b33 (table test-table [id=4a91953a6ac34dae84f582e52d00db07]) successfully deleted
I20260928 02:54:48.489475 5614 ts_tablet_manager.cc:1939] T b79a68cb1d924e30880a8f030e157b33 P 00839f70b26e453c9326babdfabbf9ca: Deleting consensus metadata
W20260928 02:54:48.489821 5108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:36089 (error 108)
I20260928 02:54:48.490655 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5315
I20260928 02:54:48.490693 5108 catalog_manager.cc:4841] TS 00839f70b26e453c9326babdfabbf9ca (127.19.139.131:33021): tablet b79a68cb1d924e30880a8f030e157b33 (table test-table [id=4a91953a6ac34dae84f582e52d00db07]) successfully deleted
I20260928 02:54:48.491050 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:48.490551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:36089 (local address 127.0.0.1:43078)
0928 02:54:48.490906 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:48.490970 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
W20260928 02:54:48.491588 5108 catalog_manager.cc:4584] TS 7c6df6c63cda48d18d248a401b84838a (127.19.139.129:36089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b79a68cb1d924e30880a8f030e157b33: Network error: Client connection negotiation failed: client connection to 127.19.139.129:36089: connect: Connection refused (error 111)
I20260928 02:54:48.495877 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5446
I20260928 02:54:48.500900 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5093
2026-09-28T02:54:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1384 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:48 dist-test-slave-5vc0 krb5kdc[5619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:48 dist-test-slave-5vc0 krb5kdc[5619](info): set up 2 sockets
Sep 28 02:54:48 dist-test-slave-5vc0 krb5kdc[5619](info): commencing operation
krb5kdc: starting...
W20260928 02:54:49.455881 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:54:49 dist-test-slave-5vc0 krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:54:49.578673 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40171
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:49.689755 5632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:49.690053 5632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:49.690114 5632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:49.694265 5632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:49.694336 5632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:49.694361 5632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:49.694382 5632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:49.694403 5632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:49.699947 5632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:49.701131 5632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:49.702204 5632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:49.711201 5640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:49.711247 5637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:49.711239 5638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:49.711323 5632 server_base.cc:1034] running on GCE node
I20260928 02:54:49.712086 5632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:49.713311 5632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:49.714483 5632 hybrid_clock.cc:648] HybridClock initialized: now 1790564089714456 us; error 36 us; skew 500 ppm
Sep 28 02:54:49 dist-test-slave-5vc0 krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564089, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:49.717772 5632 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:54:49.718791 5632 webserver.cc:460] Webserver started at http://127.19.139.190:43461/ using document root <none> and password file <none>
I20260928 02:54:49.719379 5632 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:49.719451 5632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:49.719651 5632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:49.721897 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "7d3e5173604c4819a592f5da5cecd4be"
format_stamp: "Formatted at 2026-09-28 02:54:49 on dist-test-slave-5vc0"
server_key: "0ac9b70b600e84c2ce58fbf5f08f2e09"
server_key_iv: "2cfda381bcb01459a32aaee755e03abf"
server_key_version: "encryptionkey@0"
I20260928 02:54:49.722507 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "7d3e5173604c4819a592f5da5cecd4be"
format_stamp: "Formatted at 2026-09-28 02:54:49 on dist-test-slave-5vc0"
server_key: "0ac9b70b600e84c2ce58fbf5f08f2e09"
server_key_iv: "2cfda381bcb01459a32aaee755e03abf"
server_key_version: "encryptionkey@0"
I20260928 02:54:49.726739 5632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:54:49.729588 5647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:49.731052 5632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:54:49.731177 5632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "7d3e5173604c4819a592f5da5cecd4be"
format_stamp: "Formatted at 2026-09-28 02:54:49 on dist-test-slave-5vc0"
server_key: "0ac9b70b600e84c2ce58fbf5f08f2e09"
server_key_iv: "2cfda381bcb01459a32aaee755e03abf"
server_key_version: "encryptionkey@0"
I20260928 02:54:49.731292 5632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:49.750614 5632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:49.753789 5632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:49.753986 5632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:49.761353 5632 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:40171
I20260928 02:54:49.761422 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:40171 every 8 connection(s)
I20260928 02:54:49.762338 5632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:49.765163 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:49.765316 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5632
I20260928 02:54:49.765473 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:49.765754 20014 external_mini_cluster.cc:1428] Setting key 20e39d214a24aee8e472d1dfdaa50423
I20260928 02:54:49.772334 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: Bootstrap starting.
I20260928 02:54:49.774806 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: Neither blocks nor log segments found. Creating new log.
Sep 28 02:54:49 dist-test-slave-5vc0 krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:49.775667 5700 log.cc:824] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:49.778040 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: No bootstrap required, opened a new log
I20260928 02:54:49.780550 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:49.767157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.19.139.190:40171)
0928 02:54:49.767605 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:49.767623 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:49.767666 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:54:49.768475 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:49.768502 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:49.768950 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:49.769327 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:49.770035 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:49.771519 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:49.772449 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:49.773119 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:49.773271 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:49.775781 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:49.775787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:49.775803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:49.775835 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:49.777758 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:49.778188 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:49.778194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:49.778199 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:49.778315 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:49.778565 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:49.778568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:49.778570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:49.779006 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:49.779188 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:49.779483 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:54:49.779769 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20260928 02:54:49.781853 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5173604c4819a592f5da5cecd4be" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40171 } }
I20260928 02:54:49.782080 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:49.782136 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3e5173604c4819a592f5da5cecd4be, State: Initialized, Role: FOLLOWER
I20260928 02:54:49.782567 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5173604c4819a592f5da5cecd4be" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40171 } }
I20260928 02:54:49.782703 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:49.782771 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:49.782868 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:49.783769 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5173604c4819a592f5da5cecd4be" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40171 } }
I20260928 02:54:49.784090 5700 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3e5173604c4819a592f5da5cecd4be; no voters:
I20260928 02:54:49.784409 5700 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:49.784510 5705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:49.784703 5705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [term 1 LEADER]: Becoming Leader. State: Replica: 7d3e5173604c4819a592f5da5cecd4be, State: Running, Role: LEADER
I20260928 02:54:49.785048 5705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5173604c4819a592f5da5cecd4be" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40171 } }
I20260928 02:54:49.785567 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:49.787046 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3e5173604c4819a592f5da5cecd4be. Latest consensus state: current_term: 1 leader_uuid: "7d3e5173604c4819a592f5da5cecd4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5173604c4819a592f5da5cecd4be" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40171 } } }
I20260928 02:54:49.787081 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3e5173604c4819a592f5da5cecd4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5173604c4819a592f5da5cecd4be" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40171 } } }
I20260928 02:54:49.787282 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:49.787288 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be [sys.catalog]: This master's current role is: LEADER
W20260928 02:54:49.790299 5720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:54:49.790421 5720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:54:49.790499 5719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:49.791410 5719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:49.797619 5719 catalog_manager.cc:1347] Generated new cluster ID: d6eb039eb27743afb1ea6c8a1a0bd6b4
I20260928 02:54:49.797708 5719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:49.806891 5719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:49.807807 5719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:49.813885 5719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: Generated new TSK 0
I20260928 02:54:49.814610 5719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:54:49.847765 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40171
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:49.959666 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:49.959960 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:49.960005 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:49.963994 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:49.964058 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:49.964118 5726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:49.969661 5726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:49.970858 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:49.971889 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:49.981832 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:49.981935 5726 server_base.cc:1034] running on GCE node
W20260928 02:54:49.981794 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:49.981786 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:49.982447 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:49.983088 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:49.984223 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790564089984205 us; error 29 us; skew 500 ppm
Sep 28 02:54:49 dist-test-slave-5vc0 krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564089, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:49.987107 5726 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:54:49.988117 5726 webserver.cc:460] Webserver started at http://127.19.139.129:40923/ using document root <none> and password file <none>
I20260928 02:54:49.988688 5726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:49.988755 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:49.988960 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:49.990998 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "898b0b71fc4b492ba1c91979610013f0"
format_stamp: "Formatted at 2026-09-28 02:54:49 on dist-test-slave-5vc0"
server_key: "d4250b6e54ea268aee93e31af0b5b352"
server_key_iv: "e891e48e5cdd55fc82d61e2ca22ceb19"
server_key_version: "encryptionkey@0"
I20260928 02:54:49.991531 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "898b0b71fc4b492ba1c91979610013f0"
format_stamp: "Formatted at 2026-09-28 02:54:49 on dist-test-slave-5vc0"
server_key: "d4250b6e54ea268aee93e31af0b5b352"
server_key_iv: "e891e48e5cdd55fc82d61e2ca22ceb19"
server_key_version: "encryptionkey@0"
I20260928 02:54:49.995303 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:54:49.997884 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:49.999188 5726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:49.999315 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "898b0b71fc4b492ba1c91979610013f0"
format_stamp: "Formatted at 2026-09-28 02:54:49 on dist-test-slave-5vc0"
server_key: "d4250b6e54ea268aee93e31af0b5b352"
server_key_iv: "e891e48e5cdd55fc82d61e2ca22ceb19"
server_key_version: "encryptionkey@0"
I20260928 02:54:49.999440 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:50.010586 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:50.013703 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:50.013883 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:50.014467 5726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:50.015293 5726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:50.015340 5726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.015401 5726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:50.015427 5726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.024418 5726 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:33255
I20260928 02:54:50.024439 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:33255 every 8 connection(s)
I20260928 02:54:50.025315 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:50.034107 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5726
I20260928 02:54:50.034278 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:50.034584 20014 external_mini_cluster.cc:1428] Setting key fe0f21447ec00ca0c4b9c930da9f9978
Sep 28 02:54:50 dist-test-slave-5vc0 krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564089, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:50.039170 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.027159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:57993 (local address 127.19.139.190:40171)
0928 02:54:50.027276 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.027280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.028181 (+ 901us) server_negotiation.cc:407] Connection header received
0928 02:54:50.029719 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.029723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.029781 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.029863 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:50.031028 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.031580 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.032774 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.033021 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.033111 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.035762 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:50.035765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:50.035768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:50.035792 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:50.037081 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.037625 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.037629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.037631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.037737 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.038048 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.038051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.038052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.038266 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:50.038367 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.038911 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.039046 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:54:50.040330 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.027529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40171 (local address 127.19.139.129:57993)
0928 02:54:50.027997 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.028041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.029499 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.030021 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.030028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.030386 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:50.030879 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.030893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.031697 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.031707 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.032652 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.032661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.033163 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.033167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.033327 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.033876 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:50.033893 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:50.035601 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:50.037187 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:50.037193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:50.037209 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:50.037523 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:50.037826 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:50.037828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:50.037830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:50.037973 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:50.038353 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:50.038358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:50.038630 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.038841 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.039172 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
I20260928 02:54:50.041539 5855 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40171
I20260928 02:54:50.041770 5855 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:50.042284 5855 heartbeater.cc:511] Master 127.19.139.190:40171 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:54:50.043752 5664 ts_manager.cc:194] Registered new tserver with Master: 898b0b71fc4b492ba1c91979610013f0 (127.19.139.129:33255)
I20260928 02:54:50.044962 5664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:57993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:54:50.060912 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40171
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:50.170117 5860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:50.170392 5860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:50.170433 5860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:50.174463 5860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:50.174536 5860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:50.174595 5860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:50.179992 5860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:50.181068 5860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:50.182060 5860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:50.191792 5865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:50.191910 5860 server_base.cc:1034] running on GCE node
W20260928 02:54:50.191799 5866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:50.191936 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:50.192456 5860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:50.193156 5860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:50.194308 5860 hybrid_clock.cc:648] HybridClock initialized: now 1790564090194287 us; error 35 us; skew 500 ppm
Sep 28 02:54:50 dist-test-slave-5vc0 krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564090, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:50.197180 5860 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:54:50.198087 5860 webserver.cc:460] Webserver started at http://127.19.139.130:43465/ using document root <none> and password file <none>
I20260928 02:54:50.198647 5860 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:50.198728 5860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:50.198938 5860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:50.201123 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "8e84fabddc224d83ad1eb667d391e7f4"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "fd2ce28e2f0ada68dc4e96e27f5347bd"
server_key_iv: "7019f9f808c03ffb5b4e4d34b454bfa0"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.201695 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "8e84fabddc224d83ad1eb667d391e7f4"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "fd2ce28e2f0ada68dc4e96e27f5347bd"
server_key_iv: "7019f9f808c03ffb5b4e4d34b454bfa0"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.205722 5860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:54:50.208360 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.209779 5860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:50.209920 5860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "8e84fabddc224d83ad1eb667d391e7f4"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "fd2ce28e2f0ada68dc4e96e27f5347bd"
server_key_iv: "7019f9f808c03ffb5b4e4d34b454bfa0"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.210028 5860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:50.223434 5860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:50.226612 5860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:50.226796 5860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:50.227387 5860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:50.228271 5860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:50.228320 5860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.228376 5860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:50.228405 5860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.237176 5860 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:45379
I20260928 02:54:50.237200 5988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:45379 every 8 connection(s)
I20260928 02:54:50.238142 5860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:50.247015 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5860
I20260928 02:54:50.247179 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:50.247519 20014 external_mini_cluster.cc:1428] Setting key d706c8a40520f042f664bcc855796d97
Sep 28 02:54:50 dist-test-slave-5vc0 krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564090, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:50.252645 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.240057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:52453 (local address 127.19.139.190:40171)
0928 02:54:50.240260 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.240265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.241112 (+ 847us) server_negotiation.cc:407] Connection header received
0928 02:54:50.242458 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.242461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.242511 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.242591 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:50.243909 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.244501 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.245695 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.245930 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.246008 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.248967 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:50.248971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:50.248974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:50.249002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:50.250464 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.251069 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.251072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.251073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.251157 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.251494 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.251497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.251500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.251685 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:50.251773 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.252343 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.252481 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:54:50.253520 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.240518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40171 (local address 127.19.139.130:52453)
0928 02:54:50.240956 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.240999 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.242222 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.242818 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.242826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.243216 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:50.243727 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.243742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.244634 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.244637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.245580 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.245590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.246099 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.246102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.246351 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.246968 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:50.246996 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:50.248777 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:50.250583 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:50.250591 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:50.250611 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:50.250964 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:50.251241 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:50.251244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:50.251247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:50.251402 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:50.251786 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:50.251794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:50.252063 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.252277 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.252649 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260928 02:54:50.254709 5989 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40171
I20260928 02:54:50.254956 5989 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:50.255456 5989 heartbeater.cc:511] Master 127.19.139.190:40171 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:54:50.256549 5664 ts_manager.cc:194] Registered new tserver with Master: 8e84fabddc224d83ad1eb667d391e7f4 (127.19.139.130:45379)
I20260928 02:54:50.257119 5664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:52453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:54:50.275812 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40171
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:50.387722 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:50.387992 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:50.388038 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:50.392249 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:50.392314 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:50.392396 5994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:50.397917 5994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:50.399009 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:50.400031 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:50.409607 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:50.409646 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:50.409636 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:50.409759 5994 server_base.cc:1034] running on GCE node
I20260928 02:54:50.410403 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:50.411051 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:50.412235 5994 hybrid_clock.cc:648] HybridClock initialized: now 1790564090412206 us; error 43 us; skew 500 ppm
Sep 28 02:54:50 dist-test-slave-5vc0 krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564090, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:50.415001 5994 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:54:50.415917 5994 webserver.cc:460] Webserver started at http://127.19.139.131:45035/ using document root <none> and password file <none>
I20260928 02:54:50.416517 5994 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:50.416581 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:50.416746 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:50.418769 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "15e16e5470734f91a6ee0a4305669b8b"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "9877748229f0b3ce6e3d3cc756dec9b5"
server_key_iv: "7c5f789996ae0417e6ccc3586485b897"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.419277 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "15e16e5470734f91a6ee0a4305669b8b"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "9877748229f0b3ce6e3d3cc756dec9b5"
server_key_iv: "7c5f789996ae0417e6ccc3586485b897"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.422966 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 02:54:50.425483 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.426720 5994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:50.426816 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "15e16e5470734f91a6ee0a4305669b8b"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "9877748229f0b3ce6e3d3cc756dec9b5"
server_key_iv: "7c5f789996ae0417e6ccc3586485b897"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.426904 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:50.440719 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:50.443888 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:50.444043 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:50.444649 5994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:50.445448 5994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:50.445493 5994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.445531 5994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:50.445551 5994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.453989 5994 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:41649
I20260928 02:54:50.454015 6122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:41649 every 8 connection(s)
I20260928 02:54:50.454943 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:50.462455 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 5994
I20260928 02:54:50.462600 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:50.462872 20014 external_mini_cluster.cc:1428] Setting key b25d5ea803da99e4441716ed7cf4e39f
Sep 28 02:54:50 dist-test-slave-5vc0 krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564090, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:50.468808 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.456742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:39485 (local address 127.19.139.190:40171)
0928 02:54:50.456874 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.456877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.457717 (+ 840us) server_negotiation.cc:407] Connection header received
0928 02:54:50.459072 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.459076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.459138 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.459213 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:50.460507 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.461045 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.462199 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.462506 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.462616 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.465451 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:50.465455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:50.465457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:50.465484 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:50.466772 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.467259 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.467262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.467264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.467344 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.467660 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.467663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.467665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.467863 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:50.467947 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.468513 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.468646 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:54:50.469848 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.457125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:40171 (local address 127.19.139.131:39485)
0928 02:54:50.457567 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.457611 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.458851 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.459432 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.459440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.459803 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:50.460359 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.460377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.461166 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.461169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.462058 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.462068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.462775 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.462781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.462962 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.463570 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:50.463588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:50.465290 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:50.466885 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:50.466891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:50.466908 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:50.467167 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:50.467427 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:50.467430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:50.467432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:50.467576 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:50.467960 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:50.467967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:50.468281 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.468464 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.468811 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260928 02:54:50.471052 6123 heartbeater.cc:347] Connected to a master server at 127.19.139.190:40171
I20260928 02:54:50.471282 6123 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:50.471773 6123 heartbeater.cc:511] Master 127.19.139.190:40171 requested a full tablet report, sending...
I20260928 02:54:50.472878 5664 ts_manager.cc:194] Registered new tserver with Master: 15e16e5470734f91a6ee0a4305669b8b (127.19.139.131:41649)
I20260928 02:54:50.473435 5664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:39485
I20260928 02:54:50.476655 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:50.485548 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.478527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.19.139.190:40171)
0928 02:54:50.478651 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.478655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.478796 (+ 141us) server_negotiation.cc:407] Connection header received
0928 02:54:50.478902 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.478904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.478943 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.479008 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:50.479630 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.480126 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.480916 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.481126 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.481194 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.482617 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:50.482621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:50.482623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:50.482646 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:50.483926 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.484457 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.484460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.484461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.484537 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:50.484825 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:50.484827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:50.484828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:50.485014 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:50.485123 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.485302 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.485392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:54:50.488456 5664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:50.491528 5664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:50.505849 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.499780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:41649 (local address 127.0.0.1:38690)
0928 02:54:50.500683 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.500695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.500777 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.501770 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.501773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.501789 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:50.501887 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.501894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.503177 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.503180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.504466 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.504473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.505302 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.505305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.505422 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.505460 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.505566 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.505696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":82,"thread_start_us":87,"threads_started":1}
I20260928 02:54:50.505867 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.500603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.19.139.131:41649)
0928 02:54:50.501332 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.501344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.501368 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:54:50.501443 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.501448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.501647 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.501786 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:50.502013 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.503008 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.504592 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.505189 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.505303 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.505451 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.505613 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.505709 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":77,"threads_started":1}
I20260928 02:54:50.506254 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.500538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:33255 (local address 127.0.0.1:41458)
0928 02:54:50.501004 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.501016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.501100 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.501946 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.501949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.501966 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:50.502050 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.502055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.503134 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.503138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.504447 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.504456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.505699 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.505703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.505921 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.505955 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.506024 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.506099 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":141,"thread_start_us":85,"threads_started":1}
I20260928 02:54:50.506554 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.499884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.19.139.130:45379)
0928 02:54:50.500379 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.500390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.500416 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:54:50.500599 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.500605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.500844 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.500991 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:50.501518 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.502678 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.504588 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.505395 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.505520 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.505635 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.506323 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.506427 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":126,"threads_started":1}
I20260928 02:54:50.506649 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.500968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.19.139.129:33255)
0928 02:54:50.501361 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.501374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.501396 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:54:50.501469 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.501474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.501669 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.501817 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:50.502174 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.502962 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.504556 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.505395 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.505514 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.505635 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.506429 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.506522 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":88,"threads_started":1}
I20260928 02:54:50.506254 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.499735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:45379 (local address 127.0.0.1:53694)
0928 02:54:50.500203 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.500259 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.500394 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.501202 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.501208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.501233 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:50.501385 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.501393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.502824 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.502828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.504452 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.504464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.505699 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.505703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.505912 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.506042 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.506096 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.506154 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":256,"thread_start_us":82,"threads_started":1}
I20260928 02:54:50.509414 6057 tablet_service.cc:1467] Processing CreateTablet for tablet afa9853c548945d9a8c626b1161bbfd4 (DEFAULT_TABLE table=test-table [id=bc5ed83adcbf4feda3f92265a718f38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:50.509553 5789 tablet_service.cc:1467] Processing CreateTablet for tablet afa9853c548945d9a8c626b1161bbfd4 (DEFAULT_TABLE table=test-table [id=bc5ed83adcbf4feda3f92265a718f38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:50.510473 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa9853c548945d9a8c626b1161bbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:50.510514 6057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa9853c548945d9a8c626b1161bbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:50.510246 5923 tablet_service.cc:1467] Processing CreateTablet for tablet afa9853c548945d9a8c626b1161bbfd4 (DEFAULT_TABLE table=test-table [id=bc5ed83adcbf4feda3f92265a718f38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:50.511154 5923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa9853c548945d9a8c626b1161bbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:50.517076 6140 tablet_bootstrap.cc:492] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0: Bootstrap starting.
I20260928 02:54:50.518285 6141 tablet_bootstrap.cc:492] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Bootstrap starting.
I20260928 02:54:50.518424 6142 tablet_bootstrap.cc:492] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Bootstrap starting.
I20260928 02:54:50.519351 6140 tablet_bootstrap.cc:654] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:50.520470 6140 log.cc:824] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:50.520568 6142 tablet_bootstrap.cc:654] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:50.520568 6141 tablet_bootstrap.cc:654] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:50.521845 6142 log.cc:824] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:50.521855 6141 log.cc:824] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:50.522553 6140 tablet_bootstrap.cc:492] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0: No bootstrap required, opened a new log
I20260928 02:54:50.522797 6140 ts_tablet_manager.cc:1397] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:54:50.523646 6141 tablet_bootstrap.cc:492] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: No bootstrap required, opened a new log
I20260928 02:54:50.523824 6141 ts_tablet_manager.cc:1397] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 02:54:50.523888 6142 tablet_bootstrap.cc:492] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: No bootstrap required, opened a new log
I20260928 02:54:50.524065 6142 ts_tablet_manager.cc:1397] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:54:50.526652 6141 raft_consensus.cc:348] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.526871 6141 raft_consensus.cc:374] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:50.526902 6141 raft_consensus.cc:729] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e84fabddc224d83ad1eb667d391e7f4, State: Initialized, Role: FOLLOWER
I20260928 02:54:50.526757 6140 raft_consensus.cc:348] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.527000 6140 raft_consensus.cc:374] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:50.527045 6140 raft_consensus.cc:729] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898b0b71fc4b492ba1c91979610013f0, State: Initialized, Role: FOLLOWER
I20260928 02:54:50.527287 6141 consensus_queue.cc:260] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.527463 6140 consensus_queue.cc:260] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.527935 5989 heartbeater.cc:503] Master 127.19.139.190:40171 was elected leader, sending a full tablet report...
I20260928 02:54:50.527765 6142 raft_consensus.cc:348] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.528007 6142 raft_consensus.cc:374] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:50.528049 6142 raft_consensus.cc:729] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e16e5470734f91a6ee0a4305669b8b, State: Initialized, Role: FOLLOWER
I20260928 02:54:50.528328 6141 ts_tablet_manager.cc:1428] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260928 02:54:50.528612 6140 ts_tablet_manager.cc:1428] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:54:50.528836 6142 consensus_queue.cc:260] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.528991 5855 heartbeater.cc:503] Master 127.19.139.190:40171 was elected leader, sending a full tablet report...
I20260928 02:54:50.529886 6142 ts_tablet_manager.cc:1428] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:54:50.530256 6123 heartbeater.cc:503] Master 127.19.139.190:40171 was elected leader, sending a full tablet report...
I20260928 02:54:50.701174 6147 raft_consensus.cc:482] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:50.701401 6147 raft_consensus.cc:504] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.702574 6147 leader_election.cc:288] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e84fabddc224d83ad1eb667d391e7f4 (127.19.139.130:45379), 15e16e5470734f91a6ee0a4305669b8b (127.19.139.131:41649)
W20260928 02:54:50.705865 6124 tablet.cc:2369] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:50.707906 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.702951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:55965 (local address 127.19.139.130:45379)
0928 02:54:50.703097 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.703117 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.703506 (+ 389us) server_negotiation.cc:407] Connection header received
0928 02:54:50.703574 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.703577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.703635 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.703707 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:50.704222 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.704789 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.706552 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.707178 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.707307 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.707368 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.707702 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.707783 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:54:50.707918 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.703054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:47373 (local address 127.19.139.131:41649)
0928 02:54:50.703240 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.703243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.703340 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:54:50.703548 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.703552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.703608 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.703787 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:50.704253 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.704764 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.706551 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.707191 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.707307 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.707361 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.707702 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.707781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 02:54:50.707927 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.702887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:45379 (local address 127.19.139.129:55965)
0928 02:54:50.703383 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.703395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.703481 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.703762 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.703765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.703790 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:50.704067 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.704075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.704909 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.704912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.706425 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.706432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.707344 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.707347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.707531 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.707583 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.707662 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.707747 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":201,"spinlock_wait_cycles":164096,"thread_start_us":127,"threads_started":1}
I20260928 02:54:50.708092 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.702874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:41649 (local address 127.19.139.129:47373)
0928 02:54:50.703245 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.703261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:50.703384 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:50.703790 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:50.703793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:50.703805 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:50.704076 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.704082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.704909 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.704912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.706419 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:50.706432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.707323 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.707327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:50.707514 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.707582 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:54:50.707893 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:54:50.707972 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":232,"thread_start_us":95,"threads_started":1}
I20260928 02:54:50.708734 6077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa9853c548945d9a8c626b1161bbfd4" candidate_uuid: "898b0b71fc4b492ba1c91979610013f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e16e5470734f91a6ee0a4305669b8b" is_pre_election: true
I20260928 02:54:50.708745 5943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa9853c548945d9a8c626b1161bbfd4" candidate_uuid: "898b0b71fc4b492ba1c91979610013f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e84fabddc224d83ad1eb667d391e7f4" is_pre_election: true
I20260928 02:54:50.709017 6077 raft_consensus.cc:2454] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 898b0b71fc4b492ba1c91979610013f0 in term 0.
I20260928 02:54:50.709017 5943 raft_consensus.cc:2454] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 898b0b71fc4b492ba1c91979610013f0 in term 0.
I20260928 02:54:50.709476 5743 leader_election.cc:302] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e16e5470734f91a6ee0a4305669b8b, 898b0b71fc4b492ba1c91979610013f0; no voters:
I20260928 02:54:50.709738 6147 raft_consensus.cc:2781] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:50.709829 6147 raft_consensus.cc:482] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:50.709874 6147 raft_consensus.cc:3037] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:50.710901 6147 raft_consensus.cc:504] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.711228 6147 leader_election.cc:288] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [CANDIDATE]: Term 1 election: Requested vote from peers 8e84fabddc224d83ad1eb667d391e7f4 (127.19.139.130:45379), 15e16e5470734f91a6ee0a4305669b8b (127.19.139.131:41649)
I20260928 02:54:50.711632 5943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa9853c548945d9a8c626b1161bbfd4" candidate_uuid: "898b0b71fc4b492ba1c91979610013f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e84fabddc224d83ad1eb667d391e7f4"
I20260928 02:54:50.711637 6077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa9853c548945d9a8c626b1161bbfd4" candidate_uuid: "898b0b71fc4b492ba1c91979610013f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e16e5470734f91a6ee0a4305669b8b"
I20260928 02:54:50.711752 5943 raft_consensus.cc:3037] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:50.711781 6077 raft_consensus.cc:3037] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:50.712944 5943 raft_consensus.cc:2454] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 898b0b71fc4b492ba1c91979610013f0 in term 1.
I20260928 02:54:50.713029 6077 raft_consensus.cc:2454] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 898b0b71fc4b492ba1c91979610013f0 in term 1.
I20260928 02:54:50.713279 5742 leader_election.cc:302] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 898b0b71fc4b492ba1c91979610013f0, 8e84fabddc224d83ad1eb667d391e7f4; no voters:
I20260928 02:54:50.713522 6147 raft_consensus.cc:2781] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:50.713737 6147 raft_consensus.cc:686] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [term 1 LEADER]: Becoming Leader. State: Replica: 898b0b71fc4b492ba1c91979610013f0, State: Running, Role: LEADER
I20260928 02:54:50.714071 6147 consensus_queue.cc:237] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } }
I20260928 02:54:50.717286 5661 catalog_manager.cc:5495] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 898b0b71fc4b492ba1c91979610013f0 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "898b0b71fc4b492ba1c91979610013f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "898b0b71fc4b492ba1c91979610013f0" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33255 } health_report { overall_health: HEALTHY } } }
W20260928 02:54:50.739306 5990 tablet.cc:2369] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:50.749279 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.745006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.19.139.129:33255)
0928 02:54:50.745151 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:50.745157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:50.745174 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:54:50.745289 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:50.745292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:50.745363 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:50.745476 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:50.745741 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.746551 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.747431 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:50.747784 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:50.747896 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:50.748243 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:50.748791 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:50.748953 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:50.749079 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:54:50.749125 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:54:50.755870 5943 raft_consensus.cc:1261] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 1 FOLLOWER]: Refusing update from remote peer 898b0b71fc4b492ba1c91979610013f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:50.755870 6077 raft_consensus.cc:1261] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 1 FOLLOWER]: Refusing update from remote peer 898b0b71fc4b492ba1c91979610013f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:50.756610 6151 consensus_queue.cc:1035] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e16e5470734f91a6ee0a4305669b8b" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:50.756865 6147 consensus_queue.cc:1035] T afa9853c548945d9a8c626b1161bbfd4 P 898b0b71fc4b492ba1c91979610013f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e84fabddc224d83ad1eb667d391e7f4" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:50.767004 6154 mvcc.cc:205] Tried to move back new op lower bound from 7334150515728297984 to 7334150515567632384. Current Snapshot: MvccSnapshot[applied={T|T < 7334150515728297984}]
I20260928 02:54:50.771013 6156 mvcc.cc:205] Tried to move back new op lower bound from 7334150515728297984 to 7334150515567632384. Current Snapshot: MvccSnapshot[applied={T|T < 7334150515728297984}]
I20260928 02:54:50.771341 6155 mvcc.cc:205] Tried to move back new op lower bound from 7334150515728297984 to 7334150515567632384. Current Snapshot: MvccSnapshot[applied={T|T < 7334150515728297984}]
I20260928 02:54:50.780784 5664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:50.780978 5664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:50.783850 5664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d3e5173604c4819a592f5da5cecd4be: Sending DeleteTablet for 3 replicas of tablet afa9853c548945d9a8c626b1161bbfd4
I20260928 02:54:50.784837 5923 tablet_service.cc:1514] Processing DeleteTablet for tablet afa9853c548945d9a8c626b1161bbfd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:50 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:53694
I20260928 02:54:50.784969 6057 tablet_service.cc:1514] Processing DeleteTablet for tablet afa9853c548945d9a8c626b1161bbfd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:50 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:38690
I20260928 02:54:50.786177 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5726
I20260928 02:54:50.788064 6165 tablet_replica.cc:330] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: stopping tablet replica
I20260928 02:54:50.788472 6165 raft_consensus.cc:2229] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:50.789086 6165 raft_consensus.cc:2258] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:50.789489 6164 tablet_replica.cc:330] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: stopping tablet replica
I20260928 02:54:50.789893 6164 raft_consensus.cc:2229] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:50.790272 6164 raft_consensus.cc:2258] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:50.791033 6165 ts_tablet_manager.cc:1905] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:50.791750 6164 ts_tablet_manager.cc:1905] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:50.794004 6165 ts_tablet_manager.cc:1918] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:50.794145 6165 log.cc:1196] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/afa9853c548945d9a8c626b1161bbfd4
I20260928 02:54:50.794421 6164 ts_tablet_manager.cc:1918] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:50.794483 6165 ts_tablet_manager.cc:1939] T afa9853c548945d9a8c626b1161bbfd4 P 8e84fabddc224d83ad1eb667d391e7f4: Deleting consensus metadata
W20260928 02:54:50.794582 5648 connection.cc:535] client connection to 127.19.139.129:33255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:54:50.794610 6164 log.cc:1196] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/afa9853c548945d9a8c626b1161bbfd4
W20260928 02:54:50.794737 5648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:54:50.794912 6164 ts_tablet_manager.cc:1939] T afa9853c548945d9a8c626b1161bbfd4 P 15e16e5470734f91a6ee0a4305669b8b: Deleting consensus metadata
I20260928 02:54:50.796061 5648 catalog_manager.cc:4841] TS 8e84fabddc224d83ad1eb667d391e7f4 (127.19.139.130:45379): tablet afa9853c548945d9a8c626b1161bbfd4 (table test-table [id=bc5ed83adcbf4feda3f92265a718f38c]) successfully deleted
I20260928 02:54:50.796113 5649 catalog_manager.cc:4841] TS 15e16e5470734f91a6ee0a4305669b8b (127.19.139.131:41649): tablet afa9853c548945d9a8c626b1161bbfd4 (table test-table [id=bc5ed83adcbf4feda3f92265a718f38c]) successfully deleted
I20260928 02:54:50.796267 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:50.795895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:33255 (local address 127.0.0.1:54766)
0928 02:54:50.796064 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:50.796145 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:54:50.796437 5648 catalog_manager.cc:4584] TS 898b0b71fc4b492ba1c91979610013f0 (127.19.139.129:33255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afa9853c548945d9a8c626b1161bbfd4: Network error: Client connection negotiation failed: client connection to 127.19.139.129:33255: connect: Connection refused (error 111)
I20260928 02:54:50.796545 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5860
I20260928 02:54:50.801836 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5994
I20260928 02:54:50.808315 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 5632
2026-09-28T02:54:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2308 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:50Z Disabled control of system clock
I20260928 02:54:50.836951 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42165
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:42165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:54:50.947528 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:50.947825 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:50.947861 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:50.951974 6170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:50.952034 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:50.952049 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:50.952064 6170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:50.952078 6170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:50.957038 6170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:42165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:42165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:50.958150 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:50.959553 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:50.968211 6170 server_base.cc:1034] running on GCE node
W20260928 02:54:50.968199 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:50.968214 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:50.968197 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:50.969048 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:50.970172 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:50.971360 6170 hybrid_clock.cc:648] HybridClock initialized: now 1790564090971342 us; error 33 us; skew 500 ppm
I20260928 02:54:50.973377 6170 webserver.cc:460] Webserver started at http://127.19.139.190:42761/ using document root <none> and password file <none>
I20260928 02:54:50.973975 6170 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:50.974043 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:50.974244 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:50.976648 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "b4f2f52d4aee49d988dc57cf929d00e9"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "8ee15fcdbaf26ed1607316f024eb0f64"
server_key_iv: "c917f0eba225a8c8b85403c62682a8a9"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.977218 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "b4f2f52d4aee49d988dc57cf929d00e9"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "8ee15fcdbaf26ed1607316f024eb0f64"
server_key_iv: "c917f0eba225a8c8b85403c62682a8a9"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.981103 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:54:50.983608 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:50.984938 6170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:54:50.985062 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "b4f2f52d4aee49d988dc57cf929d00e9"
format_stamp: "Formatted at 2026-09-28 02:54:50 on dist-test-slave-5vc0"
server_key: "8ee15fcdbaf26ed1607316f024eb0f64"
server_key_iv: "c917f0eba225a8c8b85403c62682a8a9"
server_key_version: "encryptionkey@0"
I20260928 02:54:50.985167 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:50.996538 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:50.997298 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:50.997494 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:51.004570 6170 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:42165
I20260928 02:54:51.004586 6236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:42165 every 8 connection(s)
I20260928 02:54:51.005647 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:51.009188 6237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:51.012845 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6170
I20260928 02:54:51.012997 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:51.013324 20014 external_mini_cluster.cc:1428] Setting key a4cb75e790d844fb4a593cda0ec1254e
I20260928 02:54:51.015926 6237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9: Bootstrap starting.
I20260928 02:54:51.018321 6237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:51.019112 6237 log.cc:824] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:51.021270 6237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9: No bootstrap required, opened a new log
I20260928 02:54:51.023272 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.014977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.19.139.190:42165)
0928 02:54:51.015727 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.015742 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.015781 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:54:51.016545 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.016567 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.017025 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.017377 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:51.017833 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.018873 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.019857 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.020426 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.020580 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.021455 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:51.021460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:51.021473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:51.021886 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:51.021945 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:51.021965 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:51.022053 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.022196 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.022517 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":142,"threads_started":1}
I20260928 02:54:51.025074 6237 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42165 } }
I20260928 02:54:51.025322 6237 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:51.025398 6237 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f2f52d4aee49d988dc57cf929d00e9, State: Initialized, Role: FOLLOWER
I20260928 02:54:51.025835 6237 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42165 } }
I20260928 02:54:51.025956 6237 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:51.026019 6237 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:51.026103 6237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:51.027060 6237 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42165 } }
I20260928 02:54:51.027393 6237 leader_election.cc:302] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f2f52d4aee49d988dc57cf929d00e9; no voters:
I20260928 02:54:51.027688 6237 leader_election.cc:288] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:51.027799 6242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:51.028023 6242 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [term 1 LEADER]: Becoming Leader. State: Replica: b4f2f52d4aee49d988dc57cf929d00e9, State: Running, Role: LEADER
I20260928 02:54:51.028403 6242 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42165 } }
I20260928 02:54:51.028906 6237 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:51.030017 6244 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f2f52d4aee49d988dc57cf929d00e9. Latest consensus state: current_term: 1 leader_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42165 } } }
I20260928 02:54:51.030148 6244 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:51.030432 6243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2f52d4aee49d988dc57cf929d00e9" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 42165 } } }
I20260928 02:54:51.030529 6243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:51.030515 6251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:51.033409 6251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:51.040202 6251 catalog_manager.cc:1347] Generated new cluster ID: 1efd7a119cbe469db371657f13ca34bc
I20260928 02:54:51.040290 6251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:51.067765 6251 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:51.068967 6251 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:51.081418 6251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9: Generated new TSK 0
I20260928 02:54:51.082315 6251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:54:51.091455 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42165
--builtin_ntp_servers=127.19.139.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:54:51.208384 6261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:51.208663 6261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:51.208701 6261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:51.212771 6261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:51.212836 6261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:51.212895 6261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:51.217816 6261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:51.218964 6261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:51.220436 6261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:51.230289 6266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:51.230396 6267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:51.230288 6269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:51.230561 6261 server_base.cc:1034] running on GCE node
I20260928 02:54:51.231092 6261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:51.231732 6261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:51.232899 6261 hybrid_clock.cc:648] HybridClock initialized: now 1790564091232879 us; error 33 us; skew 500 ppm
I20260928 02:54:51.234804 6261 webserver.cc:460] Webserver started at http://127.19.139.129:34729/ using document root <none> and password file <none>
I20260928 02:54:51.235384 6261 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:51.235451 6261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:51.235637 6261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:51.237957 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "09f85e097cfc46a285eb05d78c559788"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "968d153148bc41a71b8729c8d026e800"
server_key_iv: "6601f195db716884e5aedf51153d2b06"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.238525 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "09f85e097cfc46a285eb05d78c559788"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "968d153148bc41a71b8729c8d026e800"
server_key_iv: "6601f195db716884e5aedf51153d2b06"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.242479 6261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:54:51.245220 6275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.246475 6261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:51.246577 6261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "09f85e097cfc46a285eb05d78c559788"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "968d153148bc41a71b8729c8d026e800"
server_key_iv: "6601f195db716884e5aedf51153d2b06"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.246675 6261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:51.257829 6261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:51.258504 6261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:51.258658 6261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:51.259227 6261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:51.260385 6261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:51.260439 6261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.260481 6261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:51.260495 6261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.269160 6261 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:44857
I20260928 02:54:51.269186 6388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:44857 every 8 connection(s)
I20260928 02:54:51.270099 6261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:51.277952 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6261
I20260928 02:54:51.278101 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:51.278388 20014 external_mini_cluster.cc:1428] Setting key bca73f1b62966b8d31ad03e2fa0cc22a
I20260928 02:54:51.279618 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.271911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:33919 (local address 127.19.139.190:42165)
0928 02:54:51.272079 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.272084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.273319 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 02:54:51.274644 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.274647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.274692 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.274769 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:51.275724 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.276307 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.277575 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.277873 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.277984 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.279158 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:51.279162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:51.279165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:51.279179 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:51.279202 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:51.279212 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:51.279282 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.279414 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.279475 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:54:51.280458 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42165
--builtin_ntp_servers=127.19.139.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:54:51.280627 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.272402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42165 (local address 127.19.139.129:33919)
0928 02:54:51.273156 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.273210 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.274459 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.274987 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.274995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.275081 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:51.275594 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.275607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.276474 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.276477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.277466 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.277478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.278034 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.278039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.278233 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.278841 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:51.278843 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:51.278904 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:51.278991 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:51.279293 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:51.279300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:51.279304 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.279395 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.279670 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260928 02:54:51.281848 6389 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42165
I20260928 02:54:51.282078 6389 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:51.282583 6389 heartbeater.cc:511] Master 127.19.139.190:42165 requested a full tablet report, sending...
I20260928 02:54:51.284173 6200 ts_manager.cc:194] Registered new tserver with Master: 09f85e097cfc46a285eb05d78c559788 (127.19.139.129:44857)
I20260928 02:54:51.285358 6200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.129:33919
W20260928 02:54:51.390625 6392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:51.390899 6392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:51.390941 6392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:51.394943 6392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:51.395013 6392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:51.395068 6392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:51.399879 6392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:51.401051 6392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:51.402436 6392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:51.411896 6398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:51.411896 6397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:51.411896 6400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:51.412046 6392 server_base.cc:1034] running on GCE node
I20260928 02:54:51.412607 6392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:51.413233 6392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:51.414397 6392 hybrid_clock.cc:648] HybridClock initialized: now 1790564091414379 us; error 31 us; skew 500 ppm
I20260928 02:54:51.416239 6392 webserver.cc:460] Webserver started at http://127.19.139.130:40535/ using document root <none> and password file <none>
I20260928 02:54:51.416823 6392 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:51.416893 6392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:51.417081 6392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:51.419265 6392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "ff9b3235b0d1405c8e8d683976365970"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "7697df4efe750e788df45cd821c5c57a"
server_key_iv: "41db40db32f33b3565d1e85e3a4f8131"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.419862 6392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "ff9b3235b0d1405c8e8d683976365970"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "7697df4efe750e788df45cd821c5c57a"
server_key_iv: "41db40db32f33b3565d1e85e3a4f8131"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.423779 6392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 02:54:51.426302 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.427572 6392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:54:51.427675 6392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "ff9b3235b0d1405c8e8d683976365970"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "7697df4efe750e788df45cd821c5c57a"
server_key_iv: "41db40db32f33b3565d1e85e3a4f8131"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.427789 6392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:51.439530 6392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:51.440308 6392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:51.440490 6392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:51.441083 6392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:51.442083 6392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:51.442132 6392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.442186 6392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:51.442215 6392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.450714 6392 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:40493
I20260928 02:54:51.450870 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:40493 every 8 connection(s)
I20260928 02:54:51.451675 6392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:51.457008 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6392
I20260928 02:54:51.457130 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:51.457403 20014 external_mini_cluster.cc:1428] Setting key 5cbdf564d45f2452a7de76f20befef50
I20260928 02:54:51.459486 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42165
--builtin_ntp_servers=127.19.139.148:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:54:51.461323 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.453556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:53161 (local address 127.19.139.190:42165)
0928 02:54:51.453703 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.453708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.454842 (+ 1134us) server_negotiation.cc:407] Connection header received
0928 02:54:51.456209 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.456212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.456273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.456354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:51.457325 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.457908 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.459331 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.459646 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.459748 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.460803 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:51.460807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:51.460810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:51.460835 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:51.460858 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:51.460867 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:51.460969 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.461113 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.461181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:54:51.462224 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.453985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42165 (local address 127.19.139.130:53161)
0928 02:54:51.454679 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.454731 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.455992 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.456515 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.456522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.456611 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:51.457154 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.457201 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.458031 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.458034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.459201 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.459211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.459783 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.459786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.459947 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.460541 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:51.460542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:51.460585 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:51.460652 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:51.460981 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:51.460990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:51.460997 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.461107 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.461409 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260928 02:54:51.463371 6520 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42165
I20260928 02:54:51.463660 6520 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:51.464388 6520 heartbeater.cc:511] Master 127.19.139.190:42165 requested a full tablet report, sending...
I20260928 02:54:51.465577 6200 ts_manager.cc:194] Registered new tserver with Master: ff9b3235b0d1405c8e8d683976365970 (127.19.139.130:40493)
I20260928 02:54:51.466361 6200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.130:53161
W20260928 02:54:51.570799 6523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:51.571094 6523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:51.571132 6523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:51.575208 6523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:51.575273 6523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:51.575332 6523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:51.580147 6523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:42165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:51.581283 6523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:51.582696 6523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:51.592468 6523 server_base.cc:1034] running on GCE node
W20260928 02:54:51.592429 6531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:51.592429 6528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:51.592433 6529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:51.593089 6523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:51.593825 6523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:51.594995 6523 hybrid_clock.cc:648] HybridClock initialized: now 1790564091594961 us; error 44 us; skew 500 ppm
I20260928 02:54:51.596901 6523 webserver.cc:460] Webserver started at http://127.19.139.131:35091/ using document root <none> and password file <none>
I20260928 02:54:51.597456 6523 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:51.597519 6523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:51.597707 6523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:51.599743 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "e4d4ab765e744ba58c06f3d45544c210"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "e41c3ab7540ba50bd93910c08f804e2e"
server_key_iv: "257006273176239faf991852d30d087e"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.600310 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "e4d4ab765e744ba58c06f3d45544c210"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "e41c3ab7540ba50bd93910c08f804e2e"
server_key_iv: "257006273176239faf991852d30d087e"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.603941 6523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:51.606487 6537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.607791 6523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:54:51.607897 6523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "e4d4ab765e744ba58c06f3d45544c210"
format_stamp: "Formatted at 2026-09-28 02:54:51 on dist-test-slave-5vc0"
server_key: "e41c3ab7540ba50bd93910c08f804e2e"
server_key_iv: "257006273176239faf991852d30d087e"
server_key_version: "encryptionkey@0"
I20260928 02:54:51.608016 6523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:51.629789 6523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:51.630553 6523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:51.630728 6523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:51.631314 6523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:51.632287 6523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:51.632334 6523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.632396 6523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:51.632421 6523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:51.640870 6523 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:45633
I20260928 02:54:51.640929 6650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:45633 every 8 connection(s)
I20260928 02:54:51.641821 6523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:51.646171 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6523
I20260928 02:54:51.646302 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:51.646613 20014 external_mini_cluster.cc:1428] Setting key ce36109d7e218f21f3133aeaa5aa6404
I20260928 02:54:51.651630 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.643650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:59149 (local address 127.19.139.190:42165)
0928 02:54:51.643838 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.643842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.644952 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 02:54:51.646524 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.646527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.646581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.646693 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:51.647627 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.648413 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.649655 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.649930 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.650022 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.651157 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:51.651162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:51.651165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:51.651182 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:51.651208 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:51.651220 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:51.651285 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.651453 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.651519 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 02:54:51.652462 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.644070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:42165 (local address 127.19.139.131:59149)
0928 02:54:51.644790 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.644831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.646241 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.646878 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.646887 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.646990 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:51.647493 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.647506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.648557 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.648560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.649532 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.649542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.650114 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.650118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.650279 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.650891 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:51.650893 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:51.650935 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:51.650998 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:51.651302 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:51.651309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:51.651313 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.651405 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.651631 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":118,"threads_started":1}
I20260928 02:54:51.653498 6651 heartbeater.cc:347] Connected to a master server at 127.19.139.190:42165
I20260928 02:54:51.653728 6651 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:51.654219 6651 heartbeater.cc:511] Master 127.19.139.190:42165 requested a full tablet report, sending...
I20260928 02:54:51.655277 6200 ts_manager.cc:194] Registered new tserver with Master: e4d4ab765e744ba58c06f3d45544c210 (127.19.139.131:45633)
I20260928 02:54:51.655846 6200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.131:59149
I20260928 02:54:51.660705 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:51.666532 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.662603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.19.139.190:42165)
0928 02:54:51.662750 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.662755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.662937 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:54:51.663067 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.663070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.663111 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.663177 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:51.663447 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.663958 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.664828 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.665037 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.665121 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.666039 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:51.666042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:51.666044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:51.666056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:51.666077 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:51.666084 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:51.666206 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.666351 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.666393 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:54:51.669850 6200 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:51.672897 6200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:51.688220 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.681923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.19.139.131:45633)
0928 02:54:51.682341 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.682351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.682695 (+ 344us) server_negotiation.cc:407] Connection header received
0928 02:54:51.682925 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.682931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.683181 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.683321 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:51.683646 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.684848 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.686672 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.687405 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.687515 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.687700 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.687896 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.688030 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":81,"threads_started":1}
I20260928 02:54:51.688200 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.682309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:44857 (local address 127.0.0.1:39410)
0928 02:54:51.683558 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.683571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.683649 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.684058 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.684061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.684077 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:51.684176 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.684183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.685187 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.685191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.686442 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.686452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.687514 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.687518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.687681 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.687735 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.687847 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.687996 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"mutex_wait_us":116,"thread_start_us":120,"threads_started":1}
I20260928 02:54:51.688200 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.681567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:45633 (local address 127.0.0.1:58360)
0928 02:54:51.682398 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.682413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.682740 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.683330 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.683336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.683378 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:51.683485 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.683492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.684962 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.684966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.686545 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.686556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.687514 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.687518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.687679 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.687745 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.687847 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.687996 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":361,"spinlock_wait_cycles":38016,"thread_start_us":94,"threads_started":1}
I20260928 02:54:51.689324 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.681542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:40493 (local address 127.0.0.1:35134)
0928 02:54:51.682359 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.682410 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.682607 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.683329 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.683333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.683362 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:51.683476 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.683481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.684996 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.684998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.686546 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.686556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.688938 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.688941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.689026 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.689047 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.689120 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.689187 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":113,"spinlock_wait_cycles":9344,"thread_start_us":82,"threads_started":1}
I20260928 02:54:51.689447 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.681694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.19.139.130:40493)
0928 02:54:51.682163 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.682175 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.682508 (+ 333us) server_negotiation.cc:407] Connection header received
0928 02:54:51.682800 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.682806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.683034 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.683168 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:51.683589 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.684848 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.687762 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.688590 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.688705 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.688875 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.689161 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.689245 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":134,"threads_started":1}
I20260928 02:54:51.690169 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.682758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.19.139.129:44857)
0928 02:54:51.683143 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.683154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.683735 (+ 581us) server_negotiation.cc:407] Connection header received
0928 02:54:51.683799 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.683802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.683942 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.684064 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:51.684285 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.685072 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.686556 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.687386 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.687515 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.687700 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.689914 (+ 2214us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.690020 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":93,"threads_started":1}
I20260928 02:54:51.691766 6585 tablet_service.cc:1467] Processing CreateTablet for tablet 9617a7581f6d4ff2a64992ceacdf0419 (DEFAULT_TABLE table=test-table [id=104a1ae65911434c8c1496caac21ff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:51.692022 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 9617a7581f6d4ff2a64992ceacdf0419 (DEFAULT_TABLE table=test-table [id=104a1ae65911434c8c1496caac21ff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:51.692961 6585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9617a7581f6d4ff2a64992ceacdf0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:51.692960 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9617a7581f6d4ff2a64992ceacdf0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:51.694412 6323 tablet_service.cc:1467] Processing CreateTablet for tablet 9617a7581f6d4ff2a64992ceacdf0419 (DEFAULT_TABLE table=test-table [id=104a1ae65911434c8c1496caac21ff2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:51.695231 6323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9617a7581f6d4ff2a64992ceacdf0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:51.700457 6668 tablet_bootstrap.cc:492] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Bootstrap starting.
I20260928 02:54:51.701318 6669 tablet_bootstrap.cc:492] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Bootstrap starting.
I20260928 02:54:51.701730 6670 tablet_bootstrap.cc:492] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: Bootstrap starting.
I20260928 02:54:51.703817 6670 tablet_bootstrap.cc:654] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:51.703817 6668 tablet_bootstrap.cc:654] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:51.704588 6668 log.cc:824] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:51.704587 6670 log.cc:824] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:51.706476 6670 tablet_bootstrap.cc:492] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: No bootstrap required, opened a new log
I20260928 02:54:51.706738 6670 ts_tablet_manager.cc:1397] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:51.706745 6669 tablet_bootstrap.cc:654] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:51.707595 6669 log.cc:824] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:51.708022 6668 tablet_bootstrap.cc:492] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: No bootstrap required, opened a new log
I20260928 02:54:51.708244 6668 ts_tablet_manager.cc:1397] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:54:51.709482 6669 tablet_bootstrap.cc:492] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: No bootstrap required, opened a new log
I20260928 02:54:51.709661 6669 ts_tablet_manager.cc:1397] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 02:54:51.711129 6670 raft_consensus.cc:348] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.711462 6670 raft_consensus.cc:374] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:51.711516 6670 raft_consensus.cc:729] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f85e097cfc46a285eb05d78c559788, State: Initialized, Role: FOLLOWER
I20260928 02:54:51.712055 6670 consensus_queue.cc:260] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.712507 6668 raft_consensus.cc:348] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.712828 6668 raft_consensus.cc:374] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:51.712877 6668 raft_consensus.cc:729] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9b3235b0d1405c8e8d683976365970, State: Initialized, Role: FOLLOWER
I20260928 02:54:51.712862 6389 heartbeater.cc:503] Master 127.19.139.190:42165 was elected leader, sending a full tablet report...
I20260928 02:54:51.713140 6669 raft_consensus.cc:348] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.713367 6669 raft_consensus.cc:374] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:51.713294 6668 consensus_queue.cc:260] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.713421 6669 raft_consensus.cc:729] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d4ab765e744ba58c06f3d45544c210, State: Initialized, Role: FOLLOWER
I20260928 02:54:51.713809 6669 consensus_queue.cc:260] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.714004 6520 heartbeater.cc:503] Master 127.19.139.190:42165 was elected leader, sending a full tablet report...
I20260928 02:54:51.714233 6668 ts_tablet_manager.cc:1428] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:54:51.714591 6670 ts_tablet_manager.cc:1428] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:54:51.715564 6651 heartbeater.cc:503] Master 127.19.139.190:42165 was elected leader, sending a full tablet report...
I20260928 02:54:51.715582 6669 ts_tablet_manager.cc:1428] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:54:51.730234 6674 raft_consensus.cc:482] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:51.730487 6674 raft_consensus.cc:504] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.731884 6674 leader_election.cc:288] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff9b3235b0d1405c8e8d683976365970 (127.19.139.130:40493), e4d4ab765e744ba58c06f3d45544c210 (127.19.139.131:45633)
I20260928 02:54:51.735940 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.732253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:58477 (local address 127.19.139.131:45633)
0928 02:54:51.732386 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.732391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.732739 (+ 348us) server_negotiation.cc:407] Connection header received
0928 02:54:51.732847 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.732850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.732900 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.732968 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:51.733239 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.733802 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.735028 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.735417 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.735520 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.735572 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.735761 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.735833 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:54:51.735954 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.732127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:45633 (local address 127.19.139.129:58477)
0928 02:54:51.732626 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.732641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.732752 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.732985 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.732988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.733005 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:51.733127 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.733133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.733903 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.733906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.734919 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.734929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.735521 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.735524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.735617 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.735661 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.735722 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.735794 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":56,"thread_start_us":123,"threads_started":1}
I20260928 02:54:51.735999 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.732270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:58523 (local address 127.19.139.130:40493)
0928 02:54:51.732430 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.732434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.732453 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:54:51.732560 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.732564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.732624 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.732732 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:51.732989 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.733609 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.735035 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.735422 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.735538 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.735594 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.735801 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.735875 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 02:54:51.736001 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.732114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:40493 (local address 127.19.139.129:58523)
0928 02:54:51.732354 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.732369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:51.732464 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:51.732739 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:51.732741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:51.732760 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:51.732882 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.732889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.733717 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.733720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.734910 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:51.734920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.735546 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.735550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:51.735681 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.735702 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:54:51.735782 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:54:51.735859 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":99}
I20260928 02:54:51.736819 6605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9617a7581f6d4ff2a64992ceacdf0419" candidate_uuid: "09f85e097cfc46a285eb05d78c559788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d4ab765e744ba58c06f3d45544c210" is_pre_election: true
I20260928 02:54:51.736819 6474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9617a7581f6d4ff2a64992ceacdf0419" candidate_uuid: "09f85e097cfc46a285eb05d78c559788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9b3235b0d1405c8e8d683976365970" is_pre_election: true
I20260928 02:54:51.737144 6605 raft_consensus.cc:2454] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f85e097cfc46a285eb05d78c559788 in term 0.
I20260928 02:54:51.737144 6474 raft_consensus.cc:2454] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f85e097cfc46a285eb05d78c559788 in term 0.
I20260928 02:54:51.737632 6279 leader_election.cc:302] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f85e097cfc46a285eb05d78c559788, ff9b3235b0d1405c8e8d683976365970; no voters:
I20260928 02:54:51.737888 6674 raft_consensus.cc:2781] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:51.737970 6674 raft_consensus.cc:482] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:51.738016 6674 raft_consensus.cc:3037] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:51.739003 6674 raft_consensus.cc:504] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.739333 6674 leader_election.cc:288] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [CANDIDATE]: Term 1 election: Requested vote from peers ff9b3235b0d1405c8e8d683976365970 (127.19.139.130:40493), e4d4ab765e744ba58c06f3d45544c210 (127.19.139.131:45633)
I20260928 02:54:51.739688 6474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9617a7581f6d4ff2a64992ceacdf0419" candidate_uuid: "09f85e097cfc46a285eb05d78c559788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9b3235b0d1405c8e8d683976365970"
I20260928 02:54:51.739841 6474 raft_consensus.cc:3037] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:51.739845 6605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9617a7581f6d4ff2a64992ceacdf0419" candidate_uuid: "09f85e097cfc46a285eb05d78c559788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d4ab765e744ba58c06f3d45544c210"
I20260928 02:54:51.739969 6605 raft_consensus.cc:3037] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:51.741004 6474 raft_consensus.cc:2454] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f85e097cfc46a285eb05d78c559788 in term 1.
I20260928 02:54:51.741075 6605 raft_consensus.cc:2454] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f85e097cfc46a285eb05d78c559788 in term 1.
I20260928 02:54:51.741371 6279 leader_election.cc:302] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f85e097cfc46a285eb05d78c559788, ff9b3235b0d1405c8e8d683976365970; no voters:
I20260928 02:54:51.741565 6674 raft_consensus.cc:2781] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:51.741809 6674 raft_consensus.cc:686] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [term 1 LEADER]: Becoming Leader. State: Replica: 09f85e097cfc46a285eb05d78c559788, State: Running, Role: LEADER
I20260928 02:54:51.742166 6674 consensus_queue.cc:237] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } }
I20260928 02:54:51.745328 6201 catalog_manager.cc:5495] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f85e097cfc46a285eb05d78c559788 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "09f85e097cfc46a285eb05d78c559788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f85e097cfc46a285eb05d78c559788" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 44857 } health_report { overall_health: HEALTHY } } }
W20260928 02:54:51.771220 6390 tablet.cc:2369] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:51.772418 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.768872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.19.139.129:44857)
0928 02:54:51.768997 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:51.769000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:51.769014 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:54:51.769086 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:51.769088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:51.769142 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:51.769230 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:51.769466 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.769975 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.770816 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:51.771057 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:51.771135 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:51.771520 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:51.771938 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:51.772049 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:51.772272 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:54:51.772311 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:54:51.778335 6605 raft_consensus.cc:1261] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 1 FOLLOWER]: Refusing update from remote peer 09f85e097cfc46a285eb05d78c559788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:51.778353 6474 raft_consensus.cc:1261] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 1 FOLLOWER]: Refusing update from remote peer 09f85e097cfc46a285eb05d78c559788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:51.779052 6674 consensus_queue.cc:1035] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9b3235b0d1405c8e8d683976365970" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:51.779275 6678 consensus_queue.cc:1035] T 9617a7581f6d4ff2a64992ceacdf0419 P 09f85e097cfc46a285eb05d78c559788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d4ab765e744ba58c06f3d45544c210" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:51.786700 6681 mvcc.cc:205] Tried to move back new op lower bound from 7334150519916662784 to 7334150519778500608. Current Snapshot: MvccSnapshot[applied={T|T < 7334150519916662784}]
I20260928 02:54:51.787353 6683 mvcc.cc:205] Tried to move back new op lower bound from 7334150519916662784 to 7334150519778500608. Current Snapshot: MvccSnapshot[applied={T|T < 7334150519916662784}]
I20260928 02:54:51.796128 6201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:51.796345 6201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:51.798756 6201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4f2f52d4aee49d988dc57cf929d00e9: Sending DeleteTablet for 3 replicas of tablet 9617a7581f6d4ff2a64992ceacdf0419
I20260928 02:54:51.799630 6454 tablet_service.cc:1514] Processing DeleteTablet for tablet 9617a7581f6d4ff2a64992ceacdf0419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:51 UTC) from {username='slave'} at 127.0.0.1:35134
I20260928 02:54:51.799760 6585 tablet_service.cc:1514] Processing DeleteTablet for tablet 9617a7581f6d4ff2a64992ceacdf0419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:51 UTC) from {username='slave'} at 127.0.0.1:58360
I20260928 02:54:51.799971 6323 tablet_service.cc:1514] Processing DeleteTablet for tablet 9617a7581f6d4ff2a64992ceacdf0419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:51 UTC) from {username='slave'} at 127.0.0.1:39410
I20260928 02:54:51.800253 6691 tablet_replica.cc:330] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: stopping tablet replica
I20260928 02:54:51.800390 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6261
I20260928 02:54:51.800580 6691 raft_consensus.cc:2229] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:51.800889 6691 raft_consensus.cc:2258] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:51.801267 6692 tablet_replica.cc:330] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: stopping tablet replica
I20260928 02:54:51.801550 6692 raft_consensus.cc:2229] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:51.801827 6692 raft_consensus.cc:2258] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:51.803368 6692 ts_tablet_manager.cc:1905] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:51.803369 6691 ts_tablet_manager.cc:1905] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:51.806141 6692 ts_tablet_manager.cc:1918] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:51.806141 6691 ts_tablet_manager.cc:1918] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:51.806255 6692 log.cc:1196] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/9617a7581f6d4ff2a64992ceacdf0419
I20260928 02:54:51.806594 6691 log.cc:1196] T 9617a7581f6d4ff2a64992ceacdf0419 P ff9b3235b0d1405c8e8d683976365970: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/9617a7581f6d4ff2a64992ceacdf0419
W20260928 02:54:51.806598 6186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:44857 (error 108)
I20260928 02:54:51.806734 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6392
I20260928 02:54:51.808151 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.807856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:44857 (local address 127.0.0.1:39440)
0928 02:54:51.807988 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.808062 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:44857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:54:51.808413 6692 ts_tablet_manager.cc:1939] T 9617a7581f6d4ff2a64992ceacdf0419 P e4d4ab765e744ba58c06f3d45544c210: Deleting consensus metadata
W20260928 02:54:51.808416 6186 catalog_manager.cc:4584] TS 09f85e097cfc46a285eb05d78c559788 (127.19.139.129:44857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9617a7581f6d4ff2a64992ceacdf0419: Network error: Client connection negotiation failed: client connection to 127.19.139.129:44857: connect: Connection refused (error 111)
I20260928 02:54:51.809783 6186 catalog_manager.cc:4841] TS e4d4ab765e744ba58c06f3d45544c210 (127.19.139.131:45633): tablet 9617a7581f6d4ff2a64992ceacdf0419 (table test-table [id=104a1ae65911434c8c1496caac21ff2a]) successfully deleted
I20260928 02:54:51.812223 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6523
I20260928 02:54:51.813997 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:51.812390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:40493 (local address 127.0.0.1:35142)
0928 02:54:51.813841 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:51.813906 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:40493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1286}
I20260928 02:54:51.817274 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6170
2026-09-28T02:54:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:51 dist-test-slave-5vc0 krb5kdc[6697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:51 dist-test-slave-5vc0 krb5kdc[6697](info): set up 2 sockets
Sep 28 02:54:51 dist-test-slave-5vc0 krb5kdc[6697](info): commencing operation
krb5kdc: starting...
W20260928 02:54:52.776129 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:54:52 dist-test-slave-5vc0 krb5kdc[6697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:54:52.898924 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45817
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:53.007390 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:53.007687 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:53.007730 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:53.011831 6710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:53.011917 6710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:53.011936 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:53.011953 6710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:53.011966 6710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:53.017577 6710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:53.018693 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:53.019811 6710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:53.028570 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:53.028554 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:53.028527 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:53.028748 6710 server_base.cc:1034] running on GCE node
I20260928 02:54:53.029422 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:53.030589 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:53.031754 6710 hybrid_clock.cc:648] HybridClock initialized: now 1790564093031745 us; error 38 us; skew 500 ppm
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:53.034578 6710 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:54:53.035511 6710 webserver.cc:460] Webserver started at http://127.19.139.190:44545/ using document root <none> and password file <none>
I20260928 02:54:53.036056 6710 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:53.036123 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:53.036352 6710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:53.038574 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "b6928db7f2e64441a4c950b5bba496d4"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "e2e70f6b06ee1c5a3f46cba3dc3de3af"
server_key_iv: "3f953242ed60f04c02d4a34c3378ee48"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.039134 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "b6928db7f2e64441a4c950b5bba496d4"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "e2e70f6b06ee1c5a3f46cba3dc3de3af"
server_key_iv: "3f953242ed60f04c02d4a34c3378ee48"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.042989 6710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:54:53.045732 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.047104 6710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:54:53.047233 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "b6928db7f2e64441a4c950b5bba496d4"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "e2e70f6b06ee1c5a3f46cba3dc3de3af"
server_key_iv: "3f953242ed60f04c02d4a34c3378ee48"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.047349 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:53.066288 6710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:53.069512 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:53.069700 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:53.077131 6710 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:45817
I20260928 02:54:53.077236 6777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:45817 every 8 connection(s)
I20260928 02:54:53.078389 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:53.082341 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:53.085006 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6710
I20260928 02:54:53.085132 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:53.085390 20014 external_mini_cluster.cc:1428] Setting key c8cd25412cc43670156ce189f617c985
I20260928 02:54:53.089138 6778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4: Bootstrap starting.
I20260928 02:54:53.092039 6778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:53.092873 6778 log.cc:824] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:53.094918 6778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4: No bootstrap required, opened a new log
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:53.098218 6778 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6928db7f2e64441a4c950b5bba496d4" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45817 } }
I20260928 02:54:53.098544 6778 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:53.098609 6778 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6928db7f2e64441a4c950b5bba496d4, State: Initialized, Role: FOLLOWER
I20260928 02:54:53.099113 6778 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6928db7f2e64441a4c950b5bba496d4" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45817 } }
I20260928 02:54:53.099248 6778 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:53.099313 6778 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:53.099421 6778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:53.100603 6778 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6928db7f2e64441a4c950b5bba496d4" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45817 } }
I20260928 02:54:53.100936 6778 leader_election.cc:302] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6928db7f2e64441a4c950b5bba496d4; no voters:
I20260928 02:54:53.101248 6778 leader_election.cc:288] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:53.101302 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.086829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.19.139.190:45817)
0928 02:54:53.087304 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.087325 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.087371 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:54:53.088264 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.088290 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.088744 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.089146 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:53.089843 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.091341 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.092245 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.092745 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.092900 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.095673 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:53.095679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:53.095693 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:53.095727 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:53.097934 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.098479 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.098483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.098489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.098632 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.098993 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.098997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.098999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.099497 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:53.099699 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.100056 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.100411 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260928 02:54:53.101399 6783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:53.101595 6783 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [term 1 LEADER]: Becoming Leader. State: Replica: b6928db7f2e64441a4c950b5bba496d4, State: Running, Role: LEADER
I20260928 02:54:53.101913 6783 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6928db7f2e64441a4c950b5bba496d4" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45817 } }
I20260928 02:54:53.102434 6778 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:53.103149 6785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6928db7f2e64441a4c950b5bba496d4. Latest consensus state: current_term: 1 leader_uuid: "b6928db7f2e64441a4c950b5bba496d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6928db7f2e64441a4c950b5bba496d4" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45817 } } }
I20260928 02:54:53.103271 6785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:53.103132 6784 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6928db7f2e64441a4c950b5bba496d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6928db7f2e64441a4c950b5bba496d4" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45817 } } }
I20260928 02:54:53.103518 6784 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:53.103787 6792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:53.107040 6792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:53.113076 6792 catalog_manager.cc:1347] Generated new cluster ID: 548d336128b440ebb5a96d90888a0517
I20260928 02:54:53.113163 6792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:53.127105 6792 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:53.128078 6792 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:53.134703 6792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4: Generated new TSK 0
I20260928 02:54:53.135390 6792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.139.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
Entry for principal kudu/127.19.139.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab.
I20260928 02:54:53.168354 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45817
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:53.279145 6804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:53.279422 6804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:53.279464 6804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:53.283509 6804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:53.283577 6804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:53.283635 6804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:53.288990 6804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.129.keytab
--principal=kudu/127.19.139.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:53.290006 6804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:53.290987 6804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:53.301209 6809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:53.301313 6810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:53.301213 6812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:53.301443 6804 server_base.cc:1034] running on GCE node
I20260928 02:54:53.301988 6804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:53.302709 6804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:53.303977 6804 hybrid_clock.cc:648] HybridClock initialized: now 1790564093303946 us; error 40 us; skew 500 ppm
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:53.306784 6804 init.cc:377] Logged in from keytab as kudu/127.19.139.129@KRBTEST.COM (short username kudu)
I20260928 02:54:53.307698 6804 webserver.cc:460] Webserver started at http://127.19.139.129:39953/ using document root <none> and password file <none>
I20260928 02:54:53.308292 6804 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:53.308364 6804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:53.308564 6804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:53.310604 6804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "5803eb27669d4f3990959b7a71fbcf91"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "7b694cadc6b793a5a4c12b59a42dfbb3"
server_key_iv: "9f04d3891ea0ddf94d3b5532c2a059b9"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.311151 6804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "5803eb27669d4f3990959b7a71fbcf91"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "7b694cadc6b793a5a4c12b59a42dfbb3"
server_key_iv: "9f04d3891ea0ddf94d3b5532c2a059b9"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.314960 6804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:54:53.317472 6819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.318750 6804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:53.318893 6804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "5803eb27669d4f3990959b7a71fbcf91"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "7b694cadc6b793a5a4c12b59a42dfbb3"
server_key_iv: "9f04d3891ea0ddf94d3b5532c2a059b9"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.318997 6804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:53.330736 6804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:53.333775 6804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:53.333959 6804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:53.334563 6804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:53.335390 6804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:53.335438 6804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.335489 6804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:53.335516 6804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.343849 6804 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:33627
I20260928 02:54:53.343879 6932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:33627 every 8 connection(s)
I20260928 02:54:53.344857 6804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.129@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:53.354655 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6804
I20260928 02:54:53.354770 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:53.355084 20014 external_mini_cluster.cc:1428] Setting key 51436687ec9db98f8eeb01738e07d199
I20260928 02:54:53.358506 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.346479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:40237 (local address 127.19.139.190:45817)
0928 02:54:53.346598 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.346602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.347410 (+ 808us) server_negotiation.cc:407] Connection header received
0928 02:54:53.348719 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.348723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.348781 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.348854 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:53.350191 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.350836 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.352076 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.352395 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.352489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.354989 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:53.354993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:53.354996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:53.355027 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:53.356512 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.357014 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.357017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.357019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.357100 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.357414 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.357418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.357420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.357605 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:53.357693 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.358209 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.358335 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:54:53.359354 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.346823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45817 (local address 127.19.139.129:40237)
0928 02:54:53.347255 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.347296 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.348518 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.349113 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.349120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.349482 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:53.350019 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.350033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.350956 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.350958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.351959 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.351971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.352544 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.352549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.352722 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.353273 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:53.353289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:53.354780 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:53.356624 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:53.356629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:53.356644 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:53.356920 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:53.357183 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:53.357186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:53.357188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:53.357328 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:53.357703 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:53.357710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:53.357979 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.358154 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.358500 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260928 02:54:53.360581 6933 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45817
I20260928 02:54:53.360857 6933 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:53.361477 6933 heartbeater.cc:511] Master 127.19.139.190:45817 requested a full tablet report, sending...
I20260928 02:54:53.363075 6742 ts_manager.cc:194] Registered new tserver with Master: 5803eb27669d4f3990959b7a71fbcf91 (127.19.139.129:33627)
WARNING: no policy specified for kudu/127.19.139.130@KRBTEST.COM; defaulting to no policy
I20260928 02:54:53.364260 6742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.129@KRBTEST.COM'} at 127.19.139.129:40237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
Entry for principal kudu/127.19.139.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab.
I20260928 02:54:53.381374 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45817
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:53.492318 6938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:53.492715 6938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:53.492789 6938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:53.497089 6938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:53.497151 6938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:53.497231 6938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:53.502614 6938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.130.keytab
--principal=kudu/127.19.139.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:53.503760 6938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:53.504884 6938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:53.514873 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:53.514865 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:53.514987 6938 server_base.cc:1034] running on GCE node
W20260928 02:54:53.515141 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:53.515568 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:53.516297 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:53.517463 6938 hybrid_clock.cc:648] HybridClock initialized: now 1790564093517437 us; error 36 us; skew 500 ppm
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:53.520385 6938 init.cc:377] Logged in from keytab as kudu/127.19.139.130@KRBTEST.COM (short username kudu)
I20260928 02:54:53.521342 6938 webserver.cc:460] Webserver started at http://127.19.139.130:42313/ using document root <none> and password file <none>
I20260928 02:54:53.521951 6938 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:53.522018 6938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:53.522223 6938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:53.524458 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "299f63385b2e4769ad8a8ac14ee84ad3"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "6b664424799382e5abe0a5c8a4683ef0"
server_key_iv: "31630b752044891ab666b5ca1da0e252"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.525022 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "299f63385b2e4769ad8a8ac14ee84ad3"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "6b664424799382e5abe0a5c8a4683ef0"
server_key_iv: "31630b752044891ab666b5ca1da0e252"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.529443 6938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:54:53.532125 6953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.533715 6938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:53.533845 6938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "299f63385b2e4769ad8a8ac14ee84ad3"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "6b664424799382e5abe0a5c8a4683ef0"
server_key_iv: "31630b752044891ab666b5ca1da0e252"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.533963 6938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:53.563553 6938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:53.566972 6938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:53.567166 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:53.567850 6938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:53.568806 6938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:53.568857 6938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.568912 6938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:53.568939 6938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.577644 6938 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:33233
I20260928 02:54:53.577685 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:33233 every 8 connection(s)
I20260928 02:54:53.578786 6938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:53.588019 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 6938
I20260928 02:54:53.588184 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:53.588550 20014 external_mini_cluster.cc:1428] Setting key 414c6e0e53b9a8cf81ca8fe28e4214da
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.130@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:53.592707 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.580694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:39735 (local address 127.19.139.190:45817)
0928 02:54:53.580846 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.580849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.581718 (+ 869us) server_negotiation.cc:407] Connection header received
0928 02:54:53.583117 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.583121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.583172 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.583253 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:53.584586 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.585188 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.586429 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.586660 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.586740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.589422 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:53.589425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:53.589428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:53.589453 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:53.590752 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.591236 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.591239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.591241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.591317 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.591625 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.591629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.591630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.591815 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:53.591898 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.592476 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.592583 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:54:53.594128 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.581048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45817 (local address 127.19.139.130:39735)
0928 02:54:53.581548 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.581595 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.582888 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.583414 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.583421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.583850 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:53.584447 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.584461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.585321 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.585324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.586310 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.586319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.586824 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.586828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.586995 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.587528 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:53.587544 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:53.589241 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:53.590861 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:53.590867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:53.590884 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:53.591143 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:53.591399 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:53.591402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:53.591404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:53.591544 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:53.591911 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:53.591917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:53.592207 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.592422 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.593012 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260928 02:54:53.595381 7067 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45817
I20260928 02:54:53.595618 7067 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:53.596148 7067 heartbeater.cc:511] Master 127.19.139.190:45817 requested a full tablet report, sending...
I20260928 02:54:53.597227 6742 ts_manager.cc:194] Registered new tserver with Master: 299f63385b2e4769ad8a8ac14ee84ad3 (127.19.139.130:33233)
WARNING: no policy specified for kudu/127.19.139.131@KRBTEST.COM; defaulting to no policy
I20260928 02:54:53.597816 6742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.130@KRBTEST.COM'} at 127.19.139.130:39735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
Entry for principal kudu/127.19.139.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab.
I20260928 02:54:53.615523 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45817
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:53.726109 7072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:53.726416 7072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:53.726480 7072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:53.730615 7072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:53.730687 7072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:53.730770 7072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:53.736148 7072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.131.keytab
--principal=kudu/127.19.139.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:45817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:53.737237 7072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:53.738245 7072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:53.747843 7080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:53.747787 7077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:53.747902 7072 server_base.cc:1034] running on GCE node
W20260928 02:54:53.747787 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:53.748468 7072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:53.749126 7072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:53.750293 7072 hybrid_clock.cc:648] HybridClock initialized: now 1790564093750274 us; error 33 us; skew 500 ppm
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:53.753242 7072 init.cc:377] Logged in from keytab as kudu/127.19.139.131@KRBTEST.COM (short username kudu)
I20260928 02:54:53.754187 7072 webserver.cc:460] Webserver started at http://127.19.139.131:46213/ using document root <none> and password file <none>
I20260928 02:54:53.754740 7072 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:53.754808 7072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:53.754990 7072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:53.757198 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "957a183260e64bf89420f8a61eb450d4"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "67e6d6a57bb32b6bbfe48d09affd4af1"
server_key_iv: "527b638eb6192da0568bbc033adf308b"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.757753 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "957a183260e64bf89420f8a61eb450d4"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "67e6d6a57bb32b6bbfe48d09affd4af1"
server_key_iv: "527b638eb6192da0568bbc033adf308b"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.761641 7072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:54:53.764081 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.765514 7072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:54:53.765625 7072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "957a183260e64bf89420f8a61eb450d4"
format_stamp: "Formatted at 2026-09-28 02:54:53 on dist-test-slave-5vc0"
server_key: "67e6d6a57bb32b6bbfe48d09affd4af1"
server_key_iv: "527b638eb6192da0568bbc033adf308b"
server_key_version: "encryptionkey@0"
I20260928 02:54:53.765743 7072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:53.793658 7072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:53.796877 7072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:53.797052 7072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:53.797652 7072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:53.798498 7072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:53.798547 7072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.798589 7072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:53.798617 7072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:53.807224 7072 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:46659
I20260928 02:54:53.807243 7200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:46659 every 8 connection(s)
I20260928 02:54:53.808218 7072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:53.812239 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7072
I20260928 02:54:53.812367 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:53.812681 20014 external_mini_cluster.cc:1428] Setting key 4dccfc8f5199014195cea72385d760db
Sep 28 02:54:53 dist-test-slave-5vc0 krb5kdc[6697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564093, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.131@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:54:53.821585 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.810036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:36231 (local address 127.19.139.190:45817)
0928 02:54:53.810172 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.810176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.811034 (+ 858us) server_negotiation.cc:407] Connection header received
0928 02:54:53.812493 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.812498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.812560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.812650 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:53.814007 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.814697 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.815826 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.816040 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.816113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.818461 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:53.818464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:53.818466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:53.818492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:53.819798 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.820285 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.820287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.820289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.820362 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.820667 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.820669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.820670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.820839 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:53.820908 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.821352 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.821462 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:54:53.822590 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.810419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45817 (local address 127.19.139.131:36231)
0928 02:54:53.810873 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.810918 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.812208 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.812847 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.812855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.813250 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:53.813870 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.813889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.814822 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.814827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.815724 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.815734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.816177 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.816182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.816351 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.816873 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:53.816889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:53.818300 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:53.819894 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:53.819900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:53.819913 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:53.820192 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:53.820462 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:53.820464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:53.820465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:53.820598 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:53.820906 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:53.820911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:53.821156 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.821323 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.821589 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260928 02:54:53.823800 7201 heartbeater.cc:347] Connected to a master server at 127.19.139.190:45817
I20260928 02:54:53.824033 7201 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:53.824539 7201 heartbeater.cc:511] Master 127.19.139.190:45817 requested a full tablet report, sending...
I20260928 02:54:53.825565 6742 ts_manager.cc:194] Registered new tserver with Master: 957a183260e64bf89420f8a61eb450d4 (127.19.139.131:46659)
I20260928 02:54:53.826174 6742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.139.131@KRBTEST.COM'} at 127.19.139.131:36231
I20260928 02:54:53.826535 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:53.835757 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.828751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.19.139.190:45817)
0928 02:54:53.828877 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.828881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.829059 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:54:53.829148 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.829151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.829190 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.829254 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:53.829981 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.830505 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.831271 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.831483 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.831561 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.832872 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:53.832875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:53.832878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:53.832901 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:53.834061 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.834460 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.834462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.834464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.834564 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:53.834845 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:53.834848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:53.834850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:53.835102 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:53.835214 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.835454 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.835573 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:54:53.839031 6742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:53.842227 6742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:53.856006 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.850454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:33627 (local address 127.0.0.1:35394)
0928 02:54:53.850843 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.850891 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.851011 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.852008 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.852014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.852037 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:53.852192 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.852199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.853270 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.853273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.854371 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.854382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.855480 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.855483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.855592 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.855628 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.855727 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.855852 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260928 02:54:53.856014 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.850568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.19.139.129:33627)
0928 02:54:53.851266 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.851281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.851304 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:54:53.851377 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.851383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.851651 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.851796 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:53.852311 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.853159 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.854501 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.855345 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.855470 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.855613 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.855766 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.855864 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":81,"threads_started":1}
I20260928 02:54:53.856637 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.850537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46659 (local address 127.0.0.1:43070)
0928 02:54:53.850924 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.850937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.851025 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.852789 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.852792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.852810 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:53.852894 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.852900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.854231 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.854234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.855161 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.855170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.856254 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.856258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.856354 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.856378 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.856440 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.856507 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":114,"threads_started":1}
I20260928 02:54:53.856827 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.851532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33233 (local address 127.0.0.1:45398)
0928 02:54:53.851877 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.851889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.851962 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.852627 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.852630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.852648 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:53.852737 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.852742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.854138 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.854142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.855333 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.855343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.856451 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.856455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.856542 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.856565 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.856639 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.856705 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":76,"thread_start_us":78,"threads_started":1}
I20260928 02:54:53.856905 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.851834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.19.139.130:33233)
0928 02:54:53.852201 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.852212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.852235 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:54:53.852306 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.852311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.852494 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.852635 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:53.852830 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.853973 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.855467 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.856290 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.856449 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.856574 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.856663 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.856762 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
I20260928 02:54:53.858108 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.851294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.19.139.131:46659)
0928 02:54:53.852200 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.852216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.852235 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:54:53.852306 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.852311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.852494 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.852635 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:53.853020 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.854104 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.855273 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.856092 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.856919 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.857046 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.857092 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.857183 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":124,"threads_started":1}
I20260928 02:54:53.859200 6867 tablet_service.cc:1467] Processing CreateTablet for tablet 028d27ca18ae4fae9a64945af5d2d7bb (DEFAULT_TABLE table=test-table [id=38204277af26410e84e9dbedc1b8784f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:53.859277 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 028d27ca18ae4fae9a64945af5d2d7bb (DEFAULT_TABLE table=test-table [id=38204277af26410e84e9dbedc1b8784f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:53.860248 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028d27ca18ae4fae9a64945af5d2d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:53.860247 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028d27ca18ae4fae9a64945af5d2d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:53.861999 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 028d27ca18ae4fae9a64945af5d2d7bb (DEFAULT_TABLE table=test-table [id=38204277af26410e84e9dbedc1b8784f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:53.862982 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028d27ca18ae4fae9a64945af5d2d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:53.867735 7218 tablet_bootstrap.cc:492] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Bootstrap starting.
I20260928 02:54:53.869335 7219 tablet_bootstrap.cc:492] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Bootstrap starting.
I20260928 02:54:53.870167 7220 tablet_bootstrap.cc:492] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91: Bootstrap starting.
I20260928 02:54:53.870319 7218 tablet_bootstrap.cc:654] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:53.871711 7219 tablet_bootstrap.cc:654] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:53.871865 7218 log.cc:824] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:53.872076 7220 tablet_bootstrap.cc:654] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:53.872911 7220 log.cc:824] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:53.872960 7219 log.cc:824] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:53.874053 7218 tablet_bootstrap.cc:492] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: No bootstrap required, opened a new log
I20260928 02:54:53.874230 7218 ts_tablet_manager.cc:1397] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:54:53.874703 7220 tablet_bootstrap.cc:492] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91: No bootstrap required, opened a new log
I20260928 02:54:53.874877 7220 ts_tablet_manager.cc:1397] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:54:53.877554 7219 tablet_bootstrap.cc:492] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: No bootstrap required, opened a new log
I20260928 02:54:53.877521 7218 raft_consensus.cc:348] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.877746 7219 ts_tablet_manager.cc:1397] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 02:54:53.877810 7218 raft_consensus.cc:374] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:53.877861 7218 raft_consensus.cc:729] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299f63385b2e4769ad8a8ac14ee84ad3, State: Initialized, Role: FOLLOWER
I20260928 02:54:53.878157 7220 raft_consensus.cc:348] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.878386 7220 raft_consensus.cc:374] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:53.878343 7218 consensus_queue.cc:260] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.878436 7220 raft_consensus.cc:729] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5803eb27669d4f3990959b7a71fbcf91, State: Initialized, Role: FOLLOWER
I20260928 02:54:53.878801 7220 consensus_queue.cc:260] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.879355 7218 ts_tablet_manager.cc:1428] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:53.879688 7220 ts_tablet_manager.cc:1428] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:53.879716 7067 heartbeater.cc:503] Master 127.19.139.190:45817 was elected leader, sending a full tablet report...
I20260928 02:54:53.880007 6933 heartbeater.cc:503] Master 127.19.139.190:45817 was elected leader, sending a full tablet report...
I20260928 02:54:53.881453 7219 raft_consensus.cc:348] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.881726 7219 raft_consensus.cc:374] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:53.881778 7219 raft_consensus.cc:729] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a183260e64bf89420f8a61eb450d4, State: Initialized, Role: FOLLOWER
I20260928 02:54:53.882331 7219 consensus_queue.cc:260] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.883383 7219 ts_tablet_manager.cc:1428] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260928 02:54:53.883860 7201 heartbeater.cc:503] Master 127.19.139.190:45817 was elected leader, sending a full tablet report...
I20260928 02:54:53.955260 7225 raft_consensus.cc:482] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:53.955487 7225 raft_consensus.cc:504] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.956699 7225 leader_election.cc:288] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 957a183260e64bf89420f8a61eb450d4 (127.19.139.131:46659), 299f63385b2e4769ad8a8ac14ee84ad3 (127.19.139.130:33233)
I20260928 02:54:53.960530 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.957030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:58641 (local address 127.19.139.131:46659)
0928 02:54:53.957216 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.957220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.957389 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:54:53.957587 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.957591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.957645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.957733 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:53.958012 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.958507 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.959583 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.959994 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.960100 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.960182 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.960338 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.960407 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 02:54:53.960565 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.956972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:46659 (local address 127.19.139.129:58641)
0928 02:54:53.957288 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.957301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.957416 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.957750 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.957753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.957773 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:53.957904 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.957910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.958621 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.958625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.959461 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.959470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.960104 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.960108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.960230 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.960249 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.960312 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.960392 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"spinlock_wait_cycles":13440,"thread_start_us":135,"threads_started":1}
I20260928 02:54:53.960686 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.957550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:33351 (local address 127.19.139.130:33233)
0928 02:54:53.957670 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:53.957674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:53.957792 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:54:53.957926 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:53.957930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:53.957976 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:53.958036 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:53.958310 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.958785 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.959767 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.960190 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.960303 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.960352 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:53.960505 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:54:53.960573 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:54:53.960794 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:53.957413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33233 (local address 127.19.139.129:33351)
0928 02:54:53.957703 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:53.957714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:53.957831 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:53.958063 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:53.958065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:53.958081 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:53.958193 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.958197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.958888 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.958891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.959654 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:53.959662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:53.960304 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:53.960307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:53.960403 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:53.960420 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:54:53.960482 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:54:53.960548 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260928 02:54:53.961413 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028d27ca18ae4fae9a64945af5d2d7bb" candidate_uuid: "5803eb27669d4f3990959b7a71fbcf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a183260e64bf89420f8a61eb450d4" is_pre_election: true
I20260928 02:54:53.961419 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028d27ca18ae4fae9a64945af5d2d7bb" candidate_uuid: "5803eb27669d4f3990959b7a71fbcf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" is_pre_election: true
I20260928 02:54:53.961686 7155 raft_consensus.cc:2454] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5803eb27669d4f3990959b7a71fbcf91 in term 0.
I20260928 02:54:53.961687 7021 raft_consensus.cc:2454] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5803eb27669d4f3990959b7a71fbcf91 in term 0.
I20260928 02:54:53.962121 6823 leader_election.cc:302] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299f63385b2e4769ad8a8ac14ee84ad3, 5803eb27669d4f3990959b7a71fbcf91; no voters:
I20260928 02:54:53.962395 7225 raft_consensus.cc:2781] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:53.962473 7225 raft_consensus.cc:482] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:53.962522 7225 raft_consensus.cc:3037] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:53.963629 7225 raft_consensus.cc:504] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.963989 7225 leader_election.cc:288] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [CANDIDATE]: Term 1 election: Requested vote from peers 957a183260e64bf89420f8a61eb450d4 (127.19.139.131:46659), 299f63385b2e4769ad8a8ac14ee84ad3 (127.19.139.130:33233)
I20260928 02:54:53.964356 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028d27ca18ae4fae9a64945af5d2d7bb" candidate_uuid: "5803eb27669d4f3990959b7a71fbcf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a183260e64bf89420f8a61eb450d4"
I20260928 02:54:53.964425 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028d27ca18ae4fae9a64945af5d2d7bb" candidate_uuid: "5803eb27669d4f3990959b7a71fbcf91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299f63385b2e4769ad8a8ac14ee84ad3"
I20260928 02:54:53.964498 7155 raft_consensus.cc:3037] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:53.964537 7021 raft_consensus.cc:3037] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:53.965792 7021 raft_consensus.cc:2454] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5803eb27669d4f3990959b7a71fbcf91 in term 1.
I20260928 02:54:53.965848 7155 raft_consensus.cc:2454] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5803eb27669d4f3990959b7a71fbcf91 in term 1.
I20260928 02:54:53.966081 6823 leader_election.cc:302] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299f63385b2e4769ad8a8ac14ee84ad3, 5803eb27669d4f3990959b7a71fbcf91; no voters:
I20260928 02:54:53.966230 7225 raft_consensus.cc:2781] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:53.966444 7225 raft_consensus.cc:686] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [term 1 LEADER]: Becoming Leader. State: Replica: 5803eb27669d4f3990959b7a71fbcf91, State: Running, Role: LEADER
I20260928 02:54:53.966753 7225 consensus_queue.cc:237] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } }
I20260928 02:54:53.969841 6742 catalog_manager.cc:5495] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5803eb27669d4f3990959b7a71fbcf91 (127.19.139.129). New cstate: current_term: 1 leader_uuid: "5803eb27669d4f3990959b7a71fbcf91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803eb27669d4f3990959b7a71fbcf91" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 33627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 } health_report { overall_health: UNKNOWN } } }
I20260928 02:54:54.008942 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.005508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.19.139.129:33627)
0928 02:54:54.005637 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.005642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.005657 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:54:54.005743 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.005746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.005801 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.005900 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:54.006139 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.006641 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.007473 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.007691 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.007777 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.008079 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:54.008549 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:54.008658 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.008729 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.008791 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:54:54.014815 7021 raft_consensus.cc:1261] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 1 FOLLOWER]: Refusing update from remote peer 5803eb27669d4f3990959b7a71fbcf91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:54.014815 7155 raft_consensus.cc:1261] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 1 FOLLOWER]: Refusing update from remote peer 5803eb27669d4f3990959b7a71fbcf91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:54.015527 7225 consensus_queue.cc:1035] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a183260e64bf89420f8a61eb450d4" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 46659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:54.015740 7229 consensus_queue.cc:1035] T 028d27ca18ae4fae9a64945af5d2d7bb P 5803eb27669d4f3990959b7a71fbcf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299f63385b2e4769ad8a8ac14ee84ad3" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:54.023111 7233 mvcc.cc:205] Tried to move back new op lower bound from 7334150529077596160 to 7334150528890519552. Current Snapshot: MvccSnapshot[applied={T|T < 7334150529077596160}]
I20260928 02:54:54.034451 6742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:54.034631 6742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:54.037236 6742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6928db7f2e64441a4c950b5bba496d4: Sending DeleteTablet for 3 replicas of tablet 028d27ca18ae4fae9a64945af5d2d7bb
I20260928 02:54:54.038174 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 028d27ca18ae4fae9a64945af5d2d7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:54 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:43070
I20260928 02:54:54.038174 6867 tablet_service.cc:1514] Processing DeleteTablet for tablet 028d27ca18ae4fae9a64945af5d2d7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:54 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:35394
I20260928 02:54:54.038820 7242 tablet_replica.cc:330] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: stopping tablet replica
I20260928 02:54:54.038957 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 028d27ca18ae4fae9a64945af5d2d7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:54 UTC) from {username='kudu', principal='kudu/127.19.139.190@KRBTEST.COM'} at 127.0.0.1:45398
I20260928 02:54:54.039206 7242 raft_consensus.cc:2229] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:54.039500 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6804
I20260928 02:54:54.039573 7242 raft_consensus.cc:2258] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:54.039386 7244 tablet_replica.cc:330] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: stopping tablet replica
I20260928 02:54:54.040562 7244 raft_consensus.cc:2229] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:54.040927 7244 raft_consensus.cc:2258] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:54.042614 7244 ts_tablet_manager.cc:1905] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:54.042840 7242 ts_tablet_manager.cc:1905] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:54:54.045490 6728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:33627 (error 108)
I20260928 02:54:54.045591 7244 ts_tablet_manager.cc:1918] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:54.045591 7242 ts_tablet_manager.cc:1918] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:54.045694 7242 log.cc:1196] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-2/wal/wals/028d27ca18ae4fae9a64945af5d2d7bb
I20260928 02:54:54.045837 7244 log.cc:1196] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/028d27ca18ae4fae9a64945af5d2d7bb
I20260928 02:54:54.046022 7242 ts_tablet_manager.cc:1939] T 028d27ca18ae4fae9a64945af5d2d7bb P 957a183260e64bf89420f8a61eb450d4: Deleting consensus metadata
I20260928 02:54:54.046167 7244 ts_tablet_manager.cc:1939] T 028d27ca18ae4fae9a64945af5d2d7bb P 299f63385b2e4769ad8a8ac14ee84ad3: Deleting consensus metadata
I20260928 02:54:54.046463 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6938
I20260928 02:54:54.047202 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.046624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:33627 (local address 127.0.0.1:35412)
0928 02:54:54.047031 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.047108 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:33627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323}
W20260928 02:54:54.047376 6728 catalog_manager.cc:4584] TS 5803eb27669d4f3990959b7a71fbcf91 (127.19.139.129:33627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 028d27ca18ae4fae9a64945af5d2d7bb: Network error: Client connection negotiation failed: client connection to 127.19.139.129:33627: connect: Connection refused (error 111)
I20260928 02:54:54.049041 6729 catalog_manager.cc:4841] TS 957a183260e64bf89420f8a61eb450d4 (127.19.139.131:46659): tablet 028d27ca18ae4fae9a64945af5d2d7bb (table test-table [id=38204277af26410e84e9dbedc1b8784f]) successfully deleted
I20260928 02:54:54.051817 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7072
I20260928 02:54:54.052610 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.051951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:33233 (local address 127.0.0.1:45412)
0928 02:54:54.052463 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.052519 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:33233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
I20260928 02:54:54.056552 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 6710
2026-09-28T02:54:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2239 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:54Z Disabled control of system clock
I20260928 02:54:54.086541 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:46089
--webserver_interface=127.19.139.190
--webserver_port=0
--builtin_ntp_servers=127.19.139.148:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:54:54.196710 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:54.197031 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:54.197068 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:54.201282 7249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:54.201362 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:54.201377 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:54.201404 7249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:54.201418 7249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:54.207062 7249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:46089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:54.208328 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:54.209789 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:54.218114 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.218151 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.218122 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:54.218178 7249 server_base.cc:1034] running on GCE node
I20260928 02:54:54.218926 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:54.220055 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:54.221225 7249 hybrid_clock.cc:648] HybridClock initialized: now 1790564094221216 us; error 40 us; skew 500 ppm
I20260928 02:54:54.223178 7249 webserver.cc:460] Webserver started at http://127.19.139.190:37507/ using document root <none> and password file <none>
I20260928 02:54:54.223739 7249 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:54.223799 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:54.224021 7249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:54.226107 7249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "eed489c6d5394f83b6e846014c9c44ce"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "bf87eb3b3f4413932ad746247047a993"
server_key_iv: "15c1405a316c87308e992318b753ef41"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.226663 7249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "eed489c6d5394f83b6e846014c9c44ce"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "bf87eb3b3f4413932ad746247047a993"
server_key_iv: "15c1405a316c87308e992318b753ef41"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.230773 7249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:54:54.233449 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.234809 7249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:54:54.234939 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "eed489c6d5394f83b6e846014c9c44ce"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "bf87eb3b3f4413932ad746247047a993"
server_key_iv: "15c1405a316c87308e992318b753ef41"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.235046 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:54.277065 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:54.277817 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:54.278007 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:54.284813 7249 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:46089
I20260928 02:54:54.284838 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:46089 every 8 connection(s)
I20260928 02:54:54.285789 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:54.289027 7316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:54.293246 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7249
I20260928 02:54:54.293370 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:54.293650 20014 external_mini_cluster.cc:1428] Setting key 95adc111156e39b900fd6c0e5a6d83b9
I20260928 02:54:54.294864 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce: Bootstrap starting.
I20260928 02:54:54.297322 7316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:54.298084 7316 log.cc:824] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:54.299998 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce: No bootstrap required, opened a new log
I20260928 02:54:54.303336 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.295097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.19.139.190:46089)
0928 02:54:54.295787 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.295803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.295840 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:54:54.296559 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.296580 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.297090 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.297423 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:54.297894 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.298901 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.299816 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.300408 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.300565 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.301489 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:54.301494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:54.301508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:54.301960 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:54.302021 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:54.302042 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:54.302140 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.302303 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.302572 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":131,"threads_started":1}
I20260928 02:54:54.303671 7316 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed489c6d5394f83b6e846014c9c44ce" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46089 } }
I20260928 02:54:54.303898 7316 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:54.303962 7316 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed489c6d5394f83b6e846014c9c44ce, State: Initialized, Role: FOLLOWER
I20260928 02:54:54.304438 7316 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed489c6d5394f83b6e846014c9c44ce" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46089 } }
I20260928 02:54:54.304591 7316 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:54.304654 7316 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:54.304744 7316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:54.305912 7316 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed489c6d5394f83b6e846014c9c44ce" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46089 } }
I20260928 02:54:54.306219 7316 leader_election.cc:302] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed489c6d5394f83b6e846014c9c44ce; no voters:
I20260928 02:54:54.306495 7316 leader_election.cc:288] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:54.306608 7321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:54.306842 7321 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [term 1 LEADER]: Becoming Leader. State: Replica: eed489c6d5394f83b6e846014c9c44ce, State: Running, Role: LEADER
I20260928 02:54:54.307143 7321 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed489c6d5394f83b6e846014c9c44ce" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46089 } }
I20260928 02:54:54.307643 7316 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:54.308681 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed489c6d5394f83b6e846014c9c44ce. Latest consensus state: current_term: 1 leader_uuid: "eed489c6d5394f83b6e846014c9c44ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed489c6d5394f83b6e846014c9c44ce" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46089 } } }
I20260928 02:54:54.308816 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:54.309131 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed489c6d5394f83b6e846014c9c44ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed489c6d5394f83b6e846014c9c44ce" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 46089 } } }
I20260928 02:54:54.309194 7330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:54.309227 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:54.312561 7330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:54.318524 7330 catalog_manager.cc:1347] Generated new cluster ID: 7db068d9a59744b7a7d9c3eac7560e46
I20260928 02:54:54.318615 7330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:54.326130 7330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:54.327373 7330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:54.338191 7330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce: Generated new TSK 0
I20260928 02:54:54.339193 7330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:54:54.349997 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.129:0
--local_ip_for_outbound_sockets=127.19.139.129
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46089
--builtin_ntp_servers=127.19.139.148:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:54:54.463410 7340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:54.463696 7340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:54.463733 7340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:54.467900 7340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:54.467967 7340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:54.468022 7340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.129
I20260928 02:54:54.472908 7340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.139.129
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.129
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:54.474036 7340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:54.475414 7340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:54.485183 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.485157 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.485153 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:54.485250 7340 server_base.cc:1034] running on GCE node
I20260928 02:54:54.485811 7340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:54.486523 7340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:54.487676 7340 hybrid_clock.cc:648] HybridClock initialized: now 1790564094487660 us; error 29 us; skew 500 ppm
I20260928 02:54:54.489642 7340 webserver.cc:460] Webserver started at http://127.19.139.129:43947/ using document root <none> and password file <none>
I20260928 02:54:54.490217 7340 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:54.490284 7340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:54.490475 7340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:54.492653 7340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data/instance:
uuid: "24bbc093d77f40388015e3b52213d9f4"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "3dc3eedf866649c6bd3ac9bf58a62484"
server_key_iv: "06ca4c9773ecc61f63927f44c95c25fb"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.493202 7340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance:
uuid: "24bbc093d77f40388015e3b52213d9f4"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "3dc3eedf866649c6bd3ac9bf58a62484"
server_key_iv: "06ca4c9773ecc61f63927f44c95c25fb"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.497241 7340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:54:54.499781 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.501138 7340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:54.501281 7340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal
uuid: "24bbc093d77f40388015e3b52213d9f4"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "3dc3eedf866649c6bd3ac9bf58a62484"
server_key_iv: "06ca4c9773ecc61f63927f44c95c25fb"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.501389 7340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:54.522373 7340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:54.523301 7340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:54.523537 7340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:54.524216 7340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:54.525211 7340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:54.525259 7340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.525312 7340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:54.525347 7340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.533941 7340 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.129:35639
I20260928 02:54:54.533981 7467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.129:35639 every 8 connection(s)
I20260928 02:54:54.534852 7340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:54.536972 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7340
I20260928 02:54:54.537101 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:54.537374 20014 external_mini_cluster.cc:1428] Setting key 17e9c4f5ac4c63ec9710e395728c0eae
I20260928 02:54:54.539266 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.130:0
--local_ip_for_outbound_sockets=127.19.139.130
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46089
--builtin_ntp_servers=127.19.139.148:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:54:54.544417 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.536625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.129:32807 (local address 127.19.139.190:46089)
0928 02:54:54.536761 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.536766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.537960 (+ 1194us) server_negotiation.cc:407] Connection header received
0928 02:54:54.539517 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.539521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.539569 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.539639 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:54.540603 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.541130 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.542248 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.542550 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.542649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.543773 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:54.543778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:54.543781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:54.543796 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:54.543822 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:54.543833 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:54.543912 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.544108 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.544198 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:54:54.545252 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.537039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46089 (local address 127.19.139.129:32807)
0928 02:54:54.537802 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.537854 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:54.539225 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:54.539833 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:54.539840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:54.539937 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:54.540473 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.540488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.541249 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.541252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.542133 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.542141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.542703 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.542706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.542872 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.543478 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:54.543480 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:54.543521 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:54.543604 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:54.543953 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:54.543963 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:54.543968 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:54:54.544087 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:54:54.544385 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260928 02:54:54.546530 7468 heartbeater.cc:347] Connected to a master server at 127.19.139.190:46089
I20260928 02:54:54.546804 7468 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:54.547488 7468 heartbeater.cc:511] Master 127.19.139.190:46089 requested a full tablet report, sending...
I20260928 02:54:54.549136 7280 ts_manager.cc:194] Registered new tserver with Master: 24bbc093d77f40388015e3b52213d9f4 (127.19.139.129:35639)
I20260928 02:54:54.550287 7280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.129:32807
W20260928 02:54:54.652280 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:54.652559 7471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:54.652597 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:54.656656 7471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:54.656713 7471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:54.656767 7471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.130
I20260928 02:54:54.661592 7471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.139.130
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.130
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:54.662655 7471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:54.664000 7471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:54.673403 7476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.673412 7477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.673412 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:54.673700 7471 server_base.cc:1034] running on GCE node
I20260928 02:54:54.674242 7471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:54.674911 7471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:54.676061 7471 hybrid_clock.cc:648] HybridClock initialized: now 1790564094676028 us; error 48 us; skew 500 ppm
I20260928 02:54:54.677927 7471 webserver.cc:460] Webserver started at http://127.19.139.130:42417/ using document root <none> and password file <none>
I20260928 02:54:54.678493 7471 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:54.678560 7471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:54.678761 7471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:54.680873 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data/instance:
uuid: "53cdfc25a366409581906e33f0975ae6"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "7e9cf05e62440e1d295779a0ac026592"
server_key_iv: "bb8359e80750478bb44d5864ea9a83b5"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.681430 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance:
uuid: "53cdfc25a366409581906e33f0975ae6"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "7e9cf05e62440e1d295779a0ac026592"
server_key_iv: "bb8359e80750478bb44d5864ea9a83b5"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.685106 7471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:54.687525 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.688868 7471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:54.688982 7471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal
uuid: "53cdfc25a366409581906e33f0975ae6"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "7e9cf05e62440e1d295779a0ac026592"
server_key_iv: "bb8359e80750478bb44d5864ea9a83b5"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.689078 7471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:54.728144 7471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:54.728946 7471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:54.729132 7471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:54.729745 7471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:54.730712 7471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:54.730760 7471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.730823 7471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:54.730849 7471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.739483 7471 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.130:45573
I20260928 02:54:54.739526 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.130:45573 every 8 connection(s)
I20260928 02:54:54.740461 7471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:54.747164 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7471
I20260928 02:54:54.747310 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:54.747584 20014 external_mini_cluster.cc:1428] Setting key 54b6da74486e2437037d538a86284fb8
I20260928 02:54:54.749588 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.139.131:0
--local_ip_for_outbound_sockets=127.19.139.131
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46089
--builtin_ntp_servers=127.19.139.148:37301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:54:54.749670 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.742230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.130:51007 (local address 127.19.139.190:46089)
0928 02:54:54.742392 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.742396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.743476 (+ 1080us) server_negotiation.cc:407] Connection header received
0928 02:54:54.744889 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.744893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.744949 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.745022 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:54.745909 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.746446 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.747747 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.747970 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.748058 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.749223 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:54.749227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:54.749229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:54.749242 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:54.749266 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:54.749276 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:54.749354 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.749462 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.749525 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:54:54.750620 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.742659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46089 (local address 127.19.139.130:51007)
0928 02:54:54.743321 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.743364 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:54.744679 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:54.745197 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:54.745204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:54.745296 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:54.745774 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.745786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.746562 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.746565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.747597 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.747608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.748115 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.748119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.748329 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.748971 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:54.748973 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:54.749017 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:54.749084 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:54.749352 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:54.749357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:54.749360 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:54:54.749446 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:54:54.749671 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260928 02:54:54.751809 7599 heartbeater.cc:347] Connected to a master server at 127.19.139.190:46089
I20260928 02:54:54.752024 7599 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:54.752564 7599 heartbeater.cc:511] Master 127.19.139.190:46089 requested a full tablet report, sending...
I20260928 02:54:54.753654 7280 ts_manager.cc:194] Registered new tserver with Master: 53cdfc25a366409581906e33f0975ae6 (127.19.139.130:45573)
I20260928 02:54:54.754205 7280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.130:51007
W20260928 02:54:54.861691 7602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:54.861989 7602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:54.862043 7602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:54.866231 7602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:54.866305 7602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:54.866384 7602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.139.131
I20260928 02:54:54.871325 7602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:37301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.139.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.139.131
--webserver_port=0
--tserver_master_addrs=127.19.139.190:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.139.131
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:54.872395 7602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:54.873747 7602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:54:54.883337 7602 server_base.cc:1034] running on GCE node
W20260928 02:54:54.883392 7607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.883306 7608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:54.883306 7610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:54.883937 7602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:54.884596 7602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:54.885751 7602 hybrid_clock.cc:648] HybridClock initialized: now 1790564094885730 us; error 32 us; skew 500 ppm
I20260928 02:54:54.887477 7602 webserver.cc:460] Webserver started at http://127.19.139.131:37225/ using document root <none> and password file <none>
I20260928 02:54:54.888020 7602 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:54.888083 7602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:54.888307 7602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:54.890334 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data/instance:
uuid: "2c8dee837fbc44adb32def7394ae62fc"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "8b3b5851de3cd269657055becdf35e79"
server_key_iv: "683639e39f20620ed5bc837635c34c3f"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.890861 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance:
uuid: "2c8dee837fbc44adb32def7394ae62fc"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "8b3b5851de3cd269657055becdf35e79"
server_key_iv: "683639e39f20620ed5bc837635c34c3f"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.894464 7602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:54:54.896817 7616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.898047 7602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:54.898152 7602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal
uuid: "2c8dee837fbc44adb32def7394ae62fc"
format_stamp: "Formatted at 2026-09-28 02:54:54 on dist-test-slave-5vc0"
server_key: "8b3b5851de3cd269657055becdf35e79"
server_key_iv: "683639e39f20620ed5bc837635c34c3f"
server_key_version: "encryptionkey@0"
I20260928 02:54:54.898244 7602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:54.913818 7602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:54.914553 7602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:54.914736 7602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:54.915339 7602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:54.916353 7602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:54.916400 7602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.916457 7602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:54.916478 7602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:54.924746 7602 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.131:43965
I20260928 02:54:54.924824 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.131:43965 every 8 connection(s)
I20260928 02:54:54.925683 7602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:54.934727 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.927544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:48995 (local address 127.19.139.190:46089)
0928 02:54:54.927691 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.927695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.928767 (+ 1072us) server_negotiation.cc:407] Connection header received
0928 02:54:54.930172 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.930176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.930232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.930312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:54.931207 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.931742 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.932890 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.933114 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.933200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.934264 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:54.934267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:54.934269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:54.934282 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:54.934301 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:54.934309 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:54.934366 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.934508 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.934575 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:54:54.935599 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.927941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:46089 (local address 127.19.139.131:48995)
0928 02:54:54.928615 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.928655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:54.929963 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:54.930488 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:54.930495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:54.930582 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:54.931069 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.931081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.931857 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.931859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.932773 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.932782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.933251 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.933255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.933413 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.933997 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:54:54.933999 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:54.934043 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:54:54.934106 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:54.934377 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:54.934385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:54.934391 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:54:54.934497 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:54:54.934790 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":146,"threads_started":1}
I20260928 02:54:54.935952 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7602
I20260928 02:54:54.936074 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:54.936393 20014 external_mini_cluster.cc:1428] Setting key a111727bf416f8434f5a7f94e7d97453
I20260928 02:54:54.936856 7730 heartbeater.cc:347] Connected to a master server at 127.19.139.190:46089
I20260928 02:54:54.937100 7730 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:54.937700 7730 heartbeater.cc:511] Master 127.19.139.190:46089 requested a full tablet report, sending...
I20260928 02:54:54.938720 7280 ts_manager.cc:194] Registered new tserver with Master: 2c8dee837fbc44adb32def7394ae62fc (127.19.139.131:43965)
I20260928 02:54:54.939424 7280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.139.131:48995
I20260928 02:54:54.950294 20014 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:54.955636 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.952209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.19.139.190:46089)
0928 02:54:54.952327 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.952332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.952467 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:54:54.952562 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.952564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.952606 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.952672 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:54.952928 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.953469 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.954291 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.954503 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.954570 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.955276 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:54.955279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:54.955281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:54.955291 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:54.955305 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:54.955311 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:54.955357 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.955436 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.955485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:54:54.958503 7280 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:54.961572 7280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:54.975745 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.969659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.19.139.129:35639)
0928 02:54:54.970536 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.970548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.970570 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:54:54.970645 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.970650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.970850 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.971018 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:54.971298 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.972488 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.974248 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.974952 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.975073 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.975248 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.975476 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.975579 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":179,"threads_started":1}
I20260928 02:54:54.975749 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.970273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:45573 (local address 127.0.0.1:54900)
0928 02:54:54.970481 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.970492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:54.970577 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:54.971723 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:54.971726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:54.971743 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:54.971814 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.971820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.973003 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.973006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.974036 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.974045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.975087 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.975091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.975261 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.975318 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:54:54.975429 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:54:54.975568 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":13,"thread_start_us":82,"threads_started":1}
I20260928 02:54:54.975790 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.969499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:35639 (local address 127.0.0.1:57870)
0928 02:54:54.969906 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.969986 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:54.970160 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:54.971024 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:54.971030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:54.971055 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:54.971188 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.971194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.972579 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.972582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.974130 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.974141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.975087 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.975091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.975301 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.975318 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:54:54.975429 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:54:54.975568 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":261,"thread_start_us":96,"threads_started":1}
I20260928 02:54:54.976295 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.970653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.19.139.130:45573)
0928 02:54:54.971316 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.971326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.971348 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:54:54.971416 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.971420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.971594 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.971733 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:54.971925 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.972894 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.974152 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.974952 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.975073 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.975248 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.976013 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.976120 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":128,"threads_started":1}
I20260928 02:54:54.976567 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.969686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.131:43965 (local address 127.0.0.1:46610)
0928 02:54:54.970249 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:54.970261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:54.970328 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:54.971357 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:54.971360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:54.971377 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:54.971465 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.971472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.972538 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.972541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.974130 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:54.974141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.976181 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.976184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:54.976295 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.976317 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:54:54.976376 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:54:54.976451 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":129,"thread_start_us":88,"threads_started":1}
I20260928 02:54:54.976675 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:54.970250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.19.139.131:43965)
0928 02:54:54.970930 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:54.970941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:54.970964 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:54:54.971033 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:54.971038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:54.971214 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:54.971384 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:54.971549 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.972443 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.975307 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:54.976016 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:54.976182 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:54.976347 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:54.976436 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 02:54:54.976526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":137,"threads_started":1}
I20260928 02:54:54.979188 7402 tablet_service.cc:1467] Processing CreateTablet for tablet b7e3885e06804acca194b0cbbea849ee (DEFAULT_TABLE table=test-table [id=21c3aedf538945468ac4ba48c2546bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:54.979337 7533 tablet_service.cc:1467] Processing CreateTablet for tablet b7e3885e06804acca194b0cbbea849ee (DEFAULT_TABLE table=test-table [id=21c3aedf538945468ac4ba48c2546bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:54.979338 7664 tablet_service.cc:1467] Processing CreateTablet for tablet b7e3885e06804acca194b0cbbea849ee (DEFAULT_TABLE table=test-table [id=21c3aedf538945468ac4ba48c2546bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:54.980119 7402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e3885e06804acca194b0cbbea849ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:54.980335 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e3885e06804acca194b0cbbea849ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:54.980335 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e3885e06804acca194b0cbbea849ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:54.986443 7747 tablet_bootstrap.cc:492] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: Bootstrap starting.
I20260928 02:54:54.987552 7749 tablet_bootstrap.cc:492] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Bootstrap starting.
I20260928 02:54:54.988497 7748 tablet_bootstrap.cc:492] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: Bootstrap starting.
I20260928 02:54:54.988674 7747 tablet_bootstrap.cc:654] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:54.989364 7747 log.cc:824] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:54.989542 7749 tablet_bootstrap.cc:654] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:54.990342 7749 log.cc:824] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:54.990443 7748 tablet_bootstrap.cc:654] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:54.991139 7748 log.cc:824] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:54.991551 7747 tablet_bootstrap.cc:492] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: No bootstrap required, opened a new log
I20260928 02:54:54.991880 7747 ts_tablet_manager.cc:1397] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:54:54.992569 7749 tablet_bootstrap.cc:492] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: No bootstrap required, opened a new log
I20260928 02:54:54.992761 7749 ts_tablet_manager.cc:1397] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:54:54.993273 7748 tablet_bootstrap.cc:492] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: No bootstrap required, opened a new log
I20260928 02:54:54.993451 7748 ts_tablet_manager.cc:1397] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:54:54.996119 7747 raft_consensus.cc:348] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:54.996460 7747 raft_consensus.cc:374] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:54.996520 7747 raft_consensus.cc:729] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24bbc093d77f40388015e3b52213d9f4, State: Initialized, Role: FOLLOWER
I20260928 02:54:54.996366 7748 raft_consensus.cc:348] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:54.996623 7748 raft_consensus.cc:374] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:54.996677 7748 raft_consensus.cc:729] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8dee837fbc44adb32def7394ae62fc, State: Initialized, Role: FOLLOWER
I20260928 02:54:54.996749 7749 raft_consensus.cc:348] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:54.996977 7749 raft_consensus.cc:374] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:54.997028 7749 raft_consensus.cc:729] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cdfc25a366409581906e33f0975ae6, State: Initialized, Role: FOLLOWER
I20260928 02:54:54.996978 7747 consensus_queue.cc:260] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:54.997097 7748 consensus_queue.cc:260] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:54.997408 7749 consensus_queue.cc:260] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:54.997763 7730 heartbeater.cc:503] Master 127.19.139.190:46089 was elected leader, sending a full tablet report...
I20260928 02:54:54.998013 7747 ts_tablet_manager.cc:1428] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:54:54.998180 7749 ts_tablet_manager.cc:1428] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:54.998330 7748 ts_tablet_manager.cc:1428] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:54.998669 7599 heartbeater.cc:503] Master 127.19.139.190:46089 was elected leader, sending a full tablet report...
I20260928 02:54:55.000473 7468 heartbeater.cc:503] Master 127.19.139.190:46089 was elected leader, sending a full tablet report...
I20260928 02:54:55.017613 7754 raft_consensus.cc:482] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:55.017916 7754 raft_consensus.cc:504] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:55.019068 7754 leader_election.cc:288] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24bbc093d77f40388015e3b52213d9f4 (127.19.139.129:35639), 53cdfc25a366409581906e33f0975ae6 (127.19.139.130:45573)
I20260928 02:54:55.023303 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.019280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:35639 (local address 127.19.139.131:52883)
0928 02:54:55.019415 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:55.019432 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:55.019520 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:55.019841 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:55.019844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:55.019867 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:55.019996 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:55.020003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.020979 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.020981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:55.022139 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:55.022149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.022911 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.022915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:55.023006 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:55.023027 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:54:55.023095 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:54:55.023156 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:54:55.023332 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.019368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:52883 (local address 127.19.139.129:35639)
0928 02:54:55.019488 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:55.019492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:55.019505 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:54:55.019672 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:55.019675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:55.019736 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:55.019801 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:55.020135 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.020874 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.022277 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.022782 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.022897 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:55.022952 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:55.023121 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:54:55.023209 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:54:55.023713 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.019659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.139.131:43903 (local address 127.19.139.130:45573)
0928 02:54:55.019799 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:55.019803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:55.020004 (+ 201us) server_negotiation.cc:407] Connection header received
0928 02:54:55.020084 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:55.020087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:55.020139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:55.020242 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:55.020532 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.021192 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.022521 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.023122 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.023230 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:55.023271 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:55.023538 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:54:55.023603 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:54:55.024057 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.019542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:45573 (local address 127.19.139.131:43903)
0928 02:54:55.019891 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:55.019903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:55.019997 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:55.020306 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:55.020309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:55.020328 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:55.020427 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:55.020431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.021305 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.021309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:55.022400 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:55.022408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.023294 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.023297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:55.023389 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:55.023412 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:54:55.023854 (+ 442us) client_negotiation.cc:240] Negotiation successful
0928 02:54:55.023931 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260928 02:54:55.024000 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e3885e06804acca194b0cbbea849ee" candidate_uuid: "2c8dee837fbc44adb32def7394ae62fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24bbc093d77f40388015e3b52213d9f4" is_pre_election: true
I20260928 02:54:55.024370 7422 raft_consensus.cc:2454] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c8dee837fbc44adb32def7394ae62fc in term 0.
I20260928 02:54:55.024691 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e3885e06804acca194b0cbbea849ee" candidate_uuid: "2c8dee837fbc44adb32def7394ae62fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53cdfc25a366409581906e33f0975ae6" is_pre_election: true
I20260928 02:54:55.024897 7617 leader_election.cc:302] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24bbc093d77f40388015e3b52213d9f4, 2c8dee837fbc44adb32def7394ae62fc; no voters:
I20260928 02:54:55.025043 7553 raft_consensus.cc:2454] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c8dee837fbc44adb32def7394ae62fc in term 0.
I20260928 02:54:55.025164 7754 raft_consensus.cc:2781] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:55.025236 7754 raft_consensus.cc:482] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:55.025290 7754 raft_consensus.cc:3037] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:55.026352 7754 raft_consensus.cc:504] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:55.026672 7754 leader_election.cc:288] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [CANDIDATE]: Term 1 election: Requested vote from peers 24bbc093d77f40388015e3b52213d9f4 (127.19.139.129:35639), 53cdfc25a366409581906e33f0975ae6 (127.19.139.130:45573)
I20260928 02:54:55.027096 7422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e3885e06804acca194b0cbbea849ee" candidate_uuid: "2c8dee837fbc44adb32def7394ae62fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24bbc093d77f40388015e3b52213d9f4"
I20260928 02:54:55.027170 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e3885e06804acca194b0cbbea849ee" candidate_uuid: "2c8dee837fbc44adb32def7394ae62fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53cdfc25a366409581906e33f0975ae6"
I20260928 02:54:55.027216 7422 raft_consensus.cc:3037] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:55.027292 7553 raft_consensus.cc:3037] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:55.028275 7553 raft_consensus.cc:2454] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c8dee837fbc44adb32def7394ae62fc in term 1.
I20260928 02:54:55.028317 7422 raft_consensus.cc:2454] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c8dee837fbc44adb32def7394ae62fc in term 1.
I20260928 02:54:55.028559 7617 leader_election.cc:302] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24bbc093d77f40388015e3b52213d9f4, 2c8dee837fbc44adb32def7394ae62fc; no voters:
I20260928 02:54:55.028702 7754 raft_consensus.cc:2781] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:55.028904 7754 raft_consensus.cc:686] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 1 LEADER]: Becoming Leader. State: Replica: 2c8dee837fbc44adb32def7394ae62fc, State: Running, Role: LEADER
I20260928 02:54:55.029217 7754 consensus_queue.cc:237] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } }
I20260928 02:54:55.032454 7278 catalog_manager.cc:5495] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c8dee837fbc44adb32def7394ae62fc (127.19.139.131). New cstate: current_term: 1 leader_uuid: "2c8dee837fbc44adb32def7394ae62fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c8dee837fbc44adb32def7394ae62fc" member_type: VOTER last_known_addr { host: "127.19.139.131" port: 43965 } health_report { overall_health: HEALTHY } } }
W20260928 02:54:55.035956 7469 tablet.cc:2369] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:55.054811 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.051210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.19.139.131:43965)
0928 02:54:55.051326 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:55.051330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:55.051343 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:54:55.051453 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:55.051455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:55.051511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:55.051613 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:55.051915 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.052468 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.053339 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:55.053575 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:55.053653 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:55.053977 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:55.054421 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:55.054512 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:55.054618 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:54:55.054671 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:54:55.060894 7422 raft_consensus.cc:1261] T b7e3885e06804acca194b0cbbea849ee P 24bbc093d77f40388015e3b52213d9f4 [term 1 FOLLOWER]: Refusing update from remote peer 2c8dee837fbc44adb32def7394ae62fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:55.060914 7553 raft_consensus.cc:1261] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 1 FOLLOWER]: Refusing update from remote peer 2c8dee837fbc44adb32def7394ae62fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:55.061614 7757 consensus_queue.cc:1035] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "24bbc093d77f40388015e3b52213d9f4" member_type: VOTER last_known_addr { host: "127.19.139.129" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:55.061815 7754 consensus_queue.cc:1035] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "53cdfc25a366409581906e33f0975ae6" member_type: VOTER last_known_addr { host: "127.19.139.130" port: 45573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:55.069088 7761 mvcc.cc:205] Tried to move back new op lower bound from 7334150533361606656 to 7334150533242503168. Current Snapshot: MvccSnapshot[applied={T|T < 7334150533361606656}]
I20260928 02:54:55.076052 7278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:55.076221 7278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:55.078476 7278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed489c6d5394f83b6e846014c9c44ce: Sending DeleteTablet for 3 replicas of tablet b7e3885e06804acca194b0cbbea849ee
I20260928 02:54:55.079193 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e3885e06804acca194b0cbbea849ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:55 UTC) from {username='slave'} at 127.0.0.1:46610
I20260928 02:54:55.079200 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e3885e06804acca194b0cbbea849ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:55 UTC) from {username='slave'} at 127.0.0.1:54900
I20260928 02:54:55.079703 7771 tablet_replica.cc:330] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: stopping tablet replica
I20260928 02:54:55.079722 7402 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e3885e06804acca194b0cbbea849ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:55 UTC) from {username='slave'} at 127.0.0.1:57870
I20260928 02:54:55.080343 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7340
I20260928 02:54:55.080132 7771 raft_consensus.cc:2229] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:54:55.081336 7771 raft_consensus.cc:2258] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:55.081883 7770 tablet_replica.cc:330] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: stopping tablet replica
I20260928 02:54:55.082255 7770 raft_consensus.cc:2229] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:55.082634 7770 raft_consensus.cc:2258] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:55.083453 7771 ts_tablet_manager.cc:1905] T b7e3885e06804acca194b0cbbea849ee P 2c8dee837fbc44adb32def7394ae62fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:55.084115 7770 ts_tablet_manager.cc:1905] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:55.087906 7770 ts_tablet_manager.cc:1918] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:55.088079 7770 log.cc:1196] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Deleting WAL directory at /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563845681472-20014-0/minicluster-data/ts-1/wal/wals/b7e3885e06804acca194b0cbbea849ee
W20260928 02:54:55.088048 7264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.139.129:35639 (error 108)
I20260928 02:54:55.088413 7770 ts_tablet_manager.cc:1939] T b7e3885e06804acca194b0cbbea849ee P 53cdfc25a366409581906e33f0975ae6: Deleting consensus metadata
I20260928 02:54:55.088618 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7471
I20260928 02:54:55.093225 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.092891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.129:35639 (local address 127.0.0.1:57880)
0928 02:54:55.093032 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:55.093118 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.129:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 02:54:55.093418 7264 catalog_manager.cc:4584] TS 24bbc093d77f40388015e3b52213d9f4 (127.19.139.129:35639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7e3885e06804acca194b0cbbea849ee: Network error: Client connection negotiation failed: client connection to 127.19.139.129:35639: connect: Connection refused (error 111)
I20260928 02:54:55.094417 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7602
I20260928 02:54:55.095078 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:55.094564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.130:45573 (local address 127.0.0.1:54908)
0928 02:54:55.094953 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:55.095003 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.130:45573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
I20260928 02:54:55.099306 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7249
2026-09-28T02:54:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1041 ms)
[----------] 6 tests from EncryptionPolicyTest (10259 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:55 dist-test-slave-5vc0 krb5kdc[7776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:55 dist-test-slave-5vc0 krb5kdc[7776](info): set up 2 sockets
Sep 28 02:54:55 dist-test-slave-5vc0 krb5kdc[7776](info): commencing operation
krb5kdc: starting...
W20260928 02:54:56.056464 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:54:56 dist-test-slave-5vc0 krb5kdc[7776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:54:56.175597 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:56.285719 7789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:56.286028 7789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:56.286088 7789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:56.290261 7789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:56.290338 7789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:56.290365 7789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:56.290386 7789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:56.290406 7789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:56.296015 7789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:56.297230 7789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:56.298312 7789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:56.307045 7797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:56.307049 7795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:56.307050 7794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:56.307087 7789 server_base.cc:1034] running on GCE node
I20260928 02:54:56.307898 7789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:56.309077 7789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:56.310258 7789 hybrid_clock.cc:648] HybridClock initialized: now 1790564096310242 us; error 35 us; skew 500 ppm
Sep 28 02:54:56 dist-test-slave-5vc0 krb5kdc[7776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:56.313059 7789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:54:56.313979 7789 webserver.cc:460] Webserver started at http://127.0.0.1:35877/ using document root <none> and password file <none>
I20260928 02:54:56.314525 7789 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:56.314591 7789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:56.314795 7789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:56.316937 7789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "6dfc849d334f4cff86980daaf5dea6a3"
format_stamp: "Formatted at 2026-09-28 02:54:56 on dist-test-slave-5vc0"
server_key: "c3d66d97beddaff80ea291e8bb11b510"
server_key_iv: "019d799a29b1ab15d103f24838577f62"
server_key_version: "encryptionkey@0"
I20260928 02:54:56.317487 7789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "6dfc849d334f4cff86980daaf5dea6a3"
format_stamp: "Formatted at 2026-09-28 02:54:56 on dist-test-slave-5vc0"
server_key: "c3d66d97beddaff80ea291e8bb11b510"
server_key_iv: "019d799a29b1ab15d103f24838577f62"
server_key_version: "encryptionkey@0"
I20260928 02:54:56.321523 7789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:54:56.324122 7804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:56.325503 7789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:54:56.325635 7789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "6dfc849d334f4cff86980daaf5dea6a3"
format_stamp: "Formatted at 2026-09-28 02:54:56 on dist-test-slave-5vc0"
server_key: "c3d66d97beddaff80ea291e8bb11b510"
server_key_iv: "019d799a29b1ab15d103f24838577f62"
server_key_version: "encryptionkey@0"
I20260928 02:54:56.325740 7789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:56.344127 7789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:56.347465 7789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:56.347657 7789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:56.354414 7789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39831
I20260928 02:54:56.354447 7856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39831 every 8 connection(s)
I20260928 02:54:56.355448 7789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:56.358402 7857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:56.362210 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7789
I20260928 02:54:56.362334 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:56.362613 20014 external_mini_cluster.cc:1428] Setting key e9fc47bd94f785d22488bbc2913b9f3a
I20260928 02:54:56.364558 7857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3: Bootstrap starting.
I20260928 02:54:56.367377 7857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:56.368335 7857 log.cc:824] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:56.370822 7857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3: No bootstrap required, opened a new log
Sep 28 02:54:56 dist-test-slave-5vc0 krb5kdc[7776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:54:56.375128 7857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfc849d334f4cff86980daaf5dea6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260928 02:54:56.375381 7857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:56.375414 7857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfc849d334f4cff86980daaf5dea6a3, State: Initialized, Role: FOLLOWER
I20260928 02:54:56.375880 7857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfc849d334f4cff86980daaf5dea6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260928 02:54:56.376017 7857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:56.376086 7857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:56.376215 7857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:56.377164 7857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfc849d334f4cff86980daaf5dea6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260928 02:54:56.377507 7857 leader_election.cc:302] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dfc849d334f4cff86980daaf5dea6a3; no voters:
I20260928 02:54:56.377857 7857 leader_election.cc:288] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:56.377934 7862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:56.378137 7862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [term 1 LEADER]: Becoming Leader. State: Replica: 6dfc849d334f4cff86980daaf5dea6a3, State: Running, Role: LEADER
I20260928 02:54:56.378525 7862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfc849d334f4cff86980daaf5dea6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260928 02:54:56.378787 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:56.364107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51905 (local address 127.0.0.1:39831)
0928 02:54:56.364592 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:56.364613 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:56.364658 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:54:56.365499 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:56.365522 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:56.365994 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:56.366378 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:56.367143 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:56.368682 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:56.369738 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:56.370418 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:56.370574 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:56.373619 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:56.373625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:56.373641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:56.373674 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:56.375447 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:56.376071 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:56.376078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:56.376085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:56.376281 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:56.376602 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:56.376606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:56.376608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:56.377018 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:56.377219 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:56.377549 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:54:56.377782 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":126,"threads_started":1}
I20260928 02:54:56.379285 7857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:56.379788 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dfc849d334f4cff86980daaf5dea6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfc849d334f4cff86980daaf5dea6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260928 02:54:56.379904 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:56.381176 7863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dfc849d334f4cff86980daaf5dea6a3. Latest consensus state: current_term: 1 leader_uuid: "6dfc849d334f4cff86980daaf5dea6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dfc849d334f4cff86980daaf5dea6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260928 02:54:56.381274 7863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:56.382771 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 02:54:56.384660 7877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:54:56.384778 7877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:54:56.384851 7866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:56.385697 7866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:56.392530 7866 catalog_manager.cc:1347] Generated new cluster ID: f278c2968a204395aa1bd563a6be43a8
I20260928 02:54:56.392601 7866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:56.394085 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:56.385019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60141 (local address 127.0.0.1:39831)
0928 02:54:56.385124 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:56.385128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:56.385444 (+ 316us) server_negotiation.cc:407] Connection header received
0928 02:54:56.385557 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:56.385560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:56.385610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:56.385698 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:56.386568 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:56.387310 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:56.388524 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:56.388896 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:56.388998 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:56.390718 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:56.390722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:56.390725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:56.390760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:56.392087 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:56.392688 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:56.392691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:56.392693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:56.392782 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:56.393207 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:56.393210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:56.393212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:56.393431 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:56.393559 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:56.393840 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:54:56.393935 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:54:56.404132 7866 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:56.405542 7866 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:56.413601 7866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dfc849d334f4cff86980daaf5dea6a3: Generated new TSK 0
I20260928 02:54:56.414564 7866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:54:56.419670 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7789
W20260928 02:54:56.424295 7774 connection.cc:535] client connection to 127.0.0.1:39831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39831): Connection reset by peer (error 104)
2026-09-28T02:54:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:56 dist-test-slave-5vc0 krb5kdc[7888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:56 dist-test-slave-5vc0 krb5kdc[7888](info): set up 2 sockets
Sep 28 02:54:56 dist-test-slave-5vc0 krb5kdc[7888](info): commencing operation
krb5kdc: starting...
W20260928 02:54:57.388036 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:54:57 dist-test-slave-5vc0 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:54:57.507891 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:57.619256 7901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:57.619551 7901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:57.619597 7901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:57.623673 7901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:57.623749 7901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:57.623767 7901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:57.623791 7901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:57.623821 7901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:57.629303 7901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:57.630436 7901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:57.631496 7901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:57.639916 7906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:57.639907 7907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:57.640053 7901 server_base.cc:1034] running on GCE node
W20260928 02:54:57.639886 7909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:57.640717 7901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:57.641829 7901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:57.642980 7901 hybrid_clock.cc:648] HybridClock initialized: now 1790564097642955 us; error 41 us; skew 500 ppm
Sep 28 02:54:57 dist-test-slave-5vc0 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:57.645648 7901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:54:57.646545 7901 webserver.cc:460] Webserver started at http://127.0.0.1:45849/ using document root <none> and password file <none>
I20260928 02:54:57.647087 7901 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:57.647152 7901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:57.647349 7901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:57.649418 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11"
format_stamp: "Formatted at 2026-09-28 02:54:57 on dist-test-slave-5vc0"
server_key: "700e2039f85f3e7bdc8860af0c526e13"
server_key_iv: "991e5f90fe4a79fc02bd2246a9975538"
server_key_version: "encryptionkey@0"
I20260928 02:54:57.649958 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11"
format_stamp: "Formatted at 2026-09-28 02:54:57 on dist-test-slave-5vc0"
server_key: "700e2039f85f3e7bdc8860af0c526e13"
server_key_iv: "991e5f90fe4a79fc02bd2246a9975538"
server_key_version: "encryptionkey@0"
I20260928 02:54:57.653678 7901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:57.656071 7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:57.657436 7901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:57.657550 7901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11"
format_stamp: "Formatted at 2026-09-28 02:54:57 on dist-test-slave-5vc0"
server_key: "700e2039f85f3e7bdc8860af0c526e13"
server_key_iv: "991e5f90fe4a79fc02bd2246a9975538"
server_key_version: "encryptionkey@0"
I20260928 02:54:57.657649 7901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:57.679522 7901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:57.682458 7901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:57.682649 7901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:57.689329 7901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39807
I20260928 02:54:57.689335 7968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39807 every 8 connection(s)
I20260928 02:54:57.690276 7901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:57.692998 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:57.694379 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 7901
I20260928 02:54:57.694518 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:57.694836 20014 external_mini_cluster.cc:1428] Setting key 5a240a13d2751451f6a24a8526784439
I20260928 02:54:57.699199 7969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11: Bootstrap starting.
I20260928 02:54:57.701869 7969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:57.702638 7969 log.cc:824] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:57.703711 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:57.696146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41141 (local address 127.0.0.1:39807)
0928 02:54:57.696586 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:57.696603 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:57.696638 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:54:57.697275 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:57.697293 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:57.697642 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:57.697925 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:57.698247 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:57.699203 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:57.700405 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:57.700906 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:57.701022 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:57.701875 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:57.701880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:57.701893 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:57.702304 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:57.702363 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:57.702386 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:57.702484 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:57.702669 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:54:57.702803 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260928 02:54:57.704792 7969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11: No bootstrap required, opened a new log
I20260928 02:54:57.706903 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:54:57.709167 7969 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260928 02:54:57.709468 7969 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:57.709530 7969 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb37f08ae64464aaae3ce7f3bc9fa11, State: Initialized, Role: FOLLOWER
I20260928 02:54:57.709973 7969 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260928 02:54:57.710115 7969 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:57.710186 7969 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:57.710281 7969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:57.711364 7969 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260928 02:54:57.711731 7969 leader_election.cc:302] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdb37f08ae64464aaae3ce7f3bc9fa11; no voters:
I20260928 02:54:57.712080 7969 leader_election.cc:288] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:57.712199 7979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:57.712478 7979 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [term 1 LEADER]: Becoming Leader. State: Replica: bdb37f08ae64464aaae3ce7f3bc9fa11, State: Running, Role: LEADER
I20260928 02:54:57.712878 7979 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } }
I20260928 02:54:57.713656 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:57.709177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35895 (local address 127.0.0.1:39807)
0928 02:54:57.709306 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:57.709311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:57.709475 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:54:57.709620 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:57.709623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:57.709668 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:57.709707 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:57.710080 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:57.710807 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:57.711688 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:57.712042 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:57.712151 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:57.713133 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:57.713137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:57.713140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:57.713154 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:57.713184 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:57.713198 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:57.713289 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:57.713442 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:54:57.713480 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:54:57.713523 7969 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:57.714159 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdb37f08ae64464aaae3ce7f3bc9fa11. Latest consensus state: current_term: 1 leader_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } } }
I20260928 02:54:57.714342 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:57.714573 7981 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb37f08ae64464aaae3ce7f3bc9fa11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39807 } } }
I20260928 02:54:57.714660 7981 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:57.714773 7985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:57.718046 7985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:57.724984 7985 catalog_manager.cc:1347] Generated new cluster ID: d809684f436143ffa2593bf1996a5f4b
I20260928 02:54:57.725078 7985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:57.754724 7985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:57.755702 7985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:57.763561 7985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdb37f08ae64464aaae3ce7f3bc9fa11: Generated new TSK 0
I20260928 02:54:57.764315 7985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:54:57.773921 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 7901
W20260928 02:54:57.778499 7886 connection.cc:535] client connection to 127.0.0.1:39807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39807): Connection reset by peer (error 104)
2026-09-28T02:54:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:57 dist-test-slave-5vc0 krb5kdc[8000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:57 dist-test-slave-5vc0 krb5kdc[8000](info): set up 2 sockets
Sep 28 02:54:57 dist-test-slave-5vc0 krb5kdc[8000](info): commencing operation
krb5kdc: starting...
W20260928 02:54:58.720274 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:54:58 dist-test-slave-5vc0 krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:54:58.841249 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:58.953787 8013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:58.954087 8013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:58.954131 8013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:58.958379 8013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:58.958451 8013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:58.958468 8013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:58.958484 8013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:58.958498 8013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:58.964003 8013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:54:58.965126 8013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:58.966187 8013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:58.974681 8018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:58.974695 8021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:58.974700 8019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:58.974898 8013 server_base.cc:1034] running on GCE node
I20260928 02:54:58.975512 8013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:58.976766 8013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:58.977950 8013 hybrid_clock.cc:648] HybridClock initialized: now 1790564098977912 us; error 41 us; skew 500 ppm
Sep 28 02:54:58 dist-test-slave-5vc0 krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:58.984910 8013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:54:58.985912 8013 webserver.cc:460] Webserver started at http://127.0.0.1:40507/ using document root <none> and password file <none>
I20260928 02:54:58.986460 8013 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:58.986529 8013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:58.986733 8013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:54:58.988844 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "f0fe3e21d153420d8518c2eca31796a7"
format_stamp: "Formatted at 2026-09-28 02:54:58 on dist-test-slave-5vc0"
server_key: "2b5dd01e0789df93f8170f3180658b2f"
server_key_iv: "3dc1ae1af09105f28d78e690985b0a1e"
server_key_version: "encryptionkey@0"
I20260928 02:54:58.989377 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "f0fe3e21d153420d8518c2eca31796a7"
format_stamp: "Formatted at 2026-09-28 02:54:58 on dist-test-slave-5vc0"
server_key: "2b5dd01e0789df93f8170f3180658b2f"
server_key_iv: "3dc1ae1af09105f28d78e690985b0a1e"
server_key_version: "encryptionkey@0"
I20260928 02:54:58.993054 8013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:54:58.995496 8028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:58.996833 8013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:54:58.996977 8013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "f0fe3e21d153420d8518c2eca31796a7"
format_stamp: "Formatted at 2026-09-28 02:54:58 on dist-test-slave-5vc0"
server_key: "2b5dd01e0789df93f8170f3180658b2f"
server_key_iv: "3dc1ae1af09105f28d78e690985b0a1e"
server_key_version: "encryptionkey@0"
I20260928 02:54:58.997076 8013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:59.001507 8013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:59.004551 8013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:59.004712 8013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:59.011468 8013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41935
I20260928 02:54:59.011495 8080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41935 every 8 connection(s)
I20260928 02:54:59.012434 8013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:59.015275 8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:59.017733 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8013
I20260928 02:54:59.017884 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:54:59.018225 20014 external_mini_cluster.cc:1428] Setting key 0177fa342da3f5b9d23d251baa4fa105
I20260928 02:54:59.021801 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7: Bootstrap starting.
I20260928 02:54:59.023428 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:59.019649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47751 (local address 127.0.0.1:41935)
0928 02:54:59.020051 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:59.020067 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:59.020102 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:54:59.020806 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:59.020826 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:59.021125 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:59.021407 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:59.022233 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:59.022237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:59.022250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:59.022259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:59.022310 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:59.022329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:59.022441 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:59.022543 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:54:59.022681 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260928 02:54:59.024415 8081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:59.025178 8081 log.cc:824] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:59.026207 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:54:59.027105 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7: No bootstrap required, opened a new log
I20260928 02:54:59.029647 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:59.028141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49463 (local address 127.0.0.1:41935)
0928 02:54:59.028284 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:59.028288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:59.028413 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:54:59.028540 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:59.028543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:59.028575 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:59.028640 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:59.029299 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:59.029303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:59.029305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:54:59.029316 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:59.029343 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:54:59.029360 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:59.029439 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:59.029501 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:54:59.029537 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:54:59.030982 8081 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe3e21d153420d8518c2eca31796a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41935 } }
I20260928 02:54:59.031212 8081 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:59.031275 8081 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0fe3e21d153420d8518c2eca31796a7, State: Initialized, Role: FOLLOWER
I20260928 02:54:59.031693 8081 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe3e21d153420d8518c2eca31796a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41935 } }
I20260928 02:54:59.031827 8081 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:59.031893 8081 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:59.031987 8081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:59.032984 8081 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe3e21d153420d8518c2eca31796a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41935 } }
I20260928 02:54:59.033309 8081 leader_election.cc:302] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0fe3e21d153420d8518c2eca31796a7; no voters:
I20260928 02:54:59.033619 8081 leader_election.cc:288] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:59.033681 8091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:59.033954 8091 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [term 1 LEADER]: Becoming Leader. State: Replica: f0fe3e21d153420d8518c2eca31796a7, State: Running, Role: LEADER
I20260928 02:54:59.034313 8091 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe3e21d153420d8518c2eca31796a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41935 } }
I20260928 02:54:59.034926 8081 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:59.035904 8093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0fe3e21d153420d8518c2eca31796a7. Latest consensus state: current_term: 1 leader_uuid: "f0fe3e21d153420d8518c2eca31796a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe3e21d153420d8518c2eca31796a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41935 } } }
I20260928 02:54:59.036043 8093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:59.036356 8092 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0fe3e21d153420d8518c2eca31796a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0fe3e21d153420d8518c2eca31796a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41935 } } }
I20260928 02:54:59.036439 8100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:59.036468 8092 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:59.039410 8100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:59.044907 8100 catalog_manager.cc:1347] Generated new cluster ID: cc4b2dc4437744c8a0a8243ba3f74852
I20260928 02:54:59.044981 8100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:59.066941 8100 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:59.067301 8100 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:59.077504 8100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0fe3e21d153420d8518c2eca31796a7: Generated new TSK 0
I20260928 02:54:59.078235 8100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:54:59.083459 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8013
2026-09-28T02:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:59 dist-test-slave-5vc0 krb5kdc[8112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:59 dist-test-slave-5vc0 krb5kdc[8112](info): set up 2 sockets
Sep 28 02:54:59 dist-test-slave-5vc0 krb5kdc[8112](info): commencing operation
krb5kdc: starting...
W20260928 02:55:00.032224 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:00 dist-test-slave-5vc0 krb5kdc[8112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:00.154836 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:00.265074 8125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:00.265393 8125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:00.265436 8125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:00.269611 8125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:00.269695 8125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:00.269713 8125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:00.269732 8125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:00.269747 8125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:00.275293 8125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:00.276521 8125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:00.277590 8125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:55:00.286157 8125 server_base.cc:1034] running on GCE node
W20260928 02:55:00.286085 8131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:00.286085 8130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:00.286094 8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:00.286927 8125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:00.288086 8125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:00.289289 8125 hybrid_clock.cc:648] HybridClock initialized: now 1790564100289266 us; error 36 us; skew 500 ppm
Sep 28 02:55:00 dist-test-slave-5vc0 krb5kdc[8112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:00.292227 8125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:00.293190 8125 webserver.cc:460] Webserver started at http://127.0.0.1:45447/ using document root <none> and password file <none>
I20260928 02:55:00.293785 8125 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:00.293859 8125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:00.294067 8125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:00.296212 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "e47663e2335147e5835597c832eae61b"
format_stamp: "Formatted at 2026-09-28 02:55:00 on dist-test-slave-5vc0"
server_key: "26d511eecf6ea2d11c6c7f3383c1a2f2"
server_key_iv: "7813d3486a9df31735e9720189c480c9"
server_key_version: "encryptionkey@0"
I20260928 02:55:00.296798 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "e47663e2335147e5835597c832eae61b"
format_stamp: "Formatted at 2026-09-28 02:55:00 on dist-test-slave-5vc0"
server_key: "26d511eecf6ea2d11c6c7f3383c1a2f2"
server_key_iv: "7813d3486a9df31735e9720189c480c9"
server_key_version: "encryptionkey@0"
I20260928 02:55:00.300797 8125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:55:00.303556 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:00.304965 8125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:00.305106 8125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "e47663e2335147e5835597c832eae61b"
format_stamp: "Formatted at 2026-09-28 02:55:00 on dist-test-slave-5vc0"
server_key: "26d511eecf6ea2d11c6c7f3383c1a2f2"
server_key_iv: "7813d3486a9df31735e9720189c480c9"
server_key_version: "encryptionkey@0"
I20260928 02:55:00.305218 8125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:00.317870 8125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:00.321156 8125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:00.321357 8125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:00.328670 8125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34193
I20260928 02:55:00.328759 8192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34193 every 8 connection(s)
I20260928 02:55:00.329701 8125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:00.331137 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8125
I20260928 02:55:00.331264 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:00.331538 20014 external_mini_cluster.cc:1428] Setting key 0cff3bc4e54488fb36465519a9eb88d8
I20260928 02:55:00.332892 8193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:00.339700 8193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b: Bootstrap starting.
Sep 28 02:55:00 dist-test-slave-5vc0 krb5kdc[8112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:55:00.342429 8193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:00.343365 8193 log.cc:824] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:00.345444 8193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b: No bootstrap required, opened a new log
I20260928 02:55:00.346701 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:00.333014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51787 (local address 127.0.0.1:34193)
0928 02:55:00.333444 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:00.333463 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:00.333504 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:55:00.334352 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:00.334378 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:00.334822 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:00.335194 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:00.335860 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:00.337398 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:00.338305 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:00.339011 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:00.339164 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:00.341876 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:00.341883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:00.341899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:00.341935 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:00.343893 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:00.344384 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:00.344390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:00.344397 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:00.344512 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:00.344784 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:00.344788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:00.344790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:00.345185 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:00.345367 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:00.345657 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:55:00.345951 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":132,"threads_started":1}
I20260928 02:55:00.349253 8193 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47663e2335147e5835597c832eae61b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 02:55:00.349507 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:00.349524 8193 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:00.349582 8193 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47663e2335147e5835597c832eae61b, State: Initialized, Role: FOLLOWER
I20260928 02:55:00.350024 8193 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47663e2335147e5835597c832eae61b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 02:55:00.350170 8193 raft_consensus.cc:388] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:00.350235 8193 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:00.350334 8193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:00.351313 8193 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47663e2335147e5835597c832eae61b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 02:55:00.351681 8193 leader_election.cc:302] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e47663e2335147e5835597c832eae61b; no voters:
I20260928 02:55:00.351989 8193 leader_election.cc:288] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:00.352083 8203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:00.352281 8203 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [term 1 LEADER]: Becoming Leader. State: Replica: e47663e2335147e5835597c832eae61b, State: Running, Role: LEADER
I20260928 02:55:00.352700 8203 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47663e2335147e5835597c832eae61b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } }
I20260928 02:55:00.353381 8193 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:00.354410 8204 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e47663e2335147e5835597c832eae61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47663e2335147e5835597c832eae61b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } } }
I20260928 02:55:00.354538 8204 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:00.354858 8205 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47663e2335147e5835597c832eae61b. Latest consensus state: current_term: 1 leader_uuid: "e47663e2335147e5835597c832eae61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47663e2335147e5835597c832eae61b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34193 } } }
I20260928 02:55:00.354961 8205 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b [sys.catalog]: This master's current role is: LEADER
W20260928 02:55:00.358726 8218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:00.358863 8218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:00.358942 8212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:00.359719 8212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:00.360369 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:00.351512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39935 (local address 127.0.0.1:34193)
0928 02:55:00.351635 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:00.351640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:00.351849 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:55:00.351980 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:00.351983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:00.352028 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:00.352091 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:00.353050 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:00.353627 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:00.354829 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:00.355148 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:00.355222 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:00.356778 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:00.356782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:00.356784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:00.356810 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:00.358357 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:00.358827 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:00.358831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:00.358833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:00.358924 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:00.359229 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:00.359232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:00.359234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:00.359482 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:00.359616 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:00.360031 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 02:55:00.360200 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:55:00.365617 8212 catalog_manager.cc:1347] Generated new cluster ID: e0f54d589bf54e2980e336767cd1d812
I20260928 02:55:00.365686 8212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:00.386078 8212 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:00.387389 8212 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:00.394979 8212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47663e2335147e5835597c832eae61b: Generated new TSK 0
I20260928 02:55:00.395700 8212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:00.404736 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8125
2026-09-28T02:55:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:00 dist-test-slave-5vc0 krb5kdc[8224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:00 dist-test-slave-5vc0 krb5kdc[8224](info): set up 2 sockets
Sep 28 02:55:00 dist-test-slave-5vc0 krb5kdc[8224](info): commencing operation
krb5kdc: starting...
W20260928 02:55:01.356093 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:01 dist-test-slave-5vc0 krb5kdc[8224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:01.479491 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:01.591717 8237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:01.592034 8237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:01.592094 8237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:01.596379 8237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:01.596481 8237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:01.596508 8237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:01.596530 8237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:01.596558 8237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:01.602286 8237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:01.603458 8237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:01.604576 8237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:01.613457 8243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:01.613477 8242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:01.613552 8237 server_base.cc:1034] running on GCE node
W20260928 02:55:01.613473 8245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:01.614297 8237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:01.615442 8237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:01.616600 8237 hybrid_clock.cc:648] HybridClock initialized: now 1790564101616553 us; error 63 us; skew 500 ppm
Sep 28 02:55:01 dist-test-slave-5vc0 krb5kdc[8224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:01.619364 8237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:01.620390 8237 webserver.cc:460] Webserver started at http://127.0.0.1:35671/ using document root <none> and password file <none>
I20260928 02:55:01.620930 8237 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:01.620990 8237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:01.621153 8237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:01.623276 8237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "7c466b12d42843219f7d86e5381ecee0"
format_stamp: "Formatted at 2026-09-28 02:55:01 on dist-test-slave-5vc0"
server_key: "0b4728ed216a24ec7377fcf95c72e98f"
server_key_iv: "2d8fec48ca37c7712bdac255a2e4406e"
server_key_version: "encryptionkey@0"
I20260928 02:55:01.623822 8237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "7c466b12d42843219f7d86e5381ecee0"
format_stamp: "Formatted at 2026-09-28 02:55:01 on dist-test-slave-5vc0"
server_key: "0b4728ed216a24ec7377fcf95c72e98f"
server_key_iv: "2d8fec48ca37c7712bdac255a2e4406e"
server_key_version: "encryptionkey@0"
I20260928 02:55:01.627494 8237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 02:55:01.629941 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:01.631245 8237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:55:01.631366 8237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "7c466b12d42843219f7d86e5381ecee0"
format_stamp: "Formatted at 2026-09-28 02:55:01 on dist-test-slave-5vc0"
server_key: "0b4728ed216a24ec7377fcf95c72e98f"
server_key_iv: "2d8fec48ca37c7712bdac255a2e4406e"
server_key_version: "encryptionkey@0"
I20260928 02:55:01.631471 8237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:01.660509 8237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:01.663873 8237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:01.664048 8237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:01.670922 8237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45391
I20260928 02:55:01.670943 8304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45391 every 8 connection(s)
I20260928 02:55:01.671908 8237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:01.674834 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:01.676456 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8237
I20260928 02:55:01.676601 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:01.676923 20014 external_mini_cluster.cc:1428] Setting key 216d02c70b400ec6595dd6d37658c3a5
I20260928 02:55:01.681071 8305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0: Bootstrap starting.
I20260928 02:55:01.683677 8305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:01.684525 8305 log.cc:824] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:01.685952 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:01.678303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41013 (local address 127.0.0.1:45391)
0928 02:55:01.678704 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:01.678720 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:01.678756 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:55:01.679376 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:01.679393 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:01.679741 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:01.680018 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:01.680376 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:01.681360 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:01.682544 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:01.683063 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:01.683189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:01.684075 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:01.684080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:01.684094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:01.684561 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:01.684629 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:01.684662 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:01.684760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:01.684916 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:55:01.685146 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260928 02:55:01.686563 8305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0: No bootstrap required, opened a new log
I20260928 02:55:01.689091 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:01.691077 8305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c466b12d42843219f7d86e5381ecee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45391 } }
I20260928 02:55:01.691385 8305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:01.691500 8305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c466b12d42843219f7d86e5381ecee0, State: Initialized, Role: FOLLOWER
I20260928 02:55:01.692024 8305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c466b12d42843219f7d86e5381ecee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45391 } }
I20260928 02:55:01.692220 8305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:01.692307 8305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:01.692414 8305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:01.693563 8305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c466b12d42843219f7d86e5381ecee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45391 } }
I20260928 02:55:01.693965 8305 leader_election.cc:302] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c466b12d42843219f7d86e5381ecee0; no voters:
I20260928 02:55:01.694334 8305 leader_election.cc:288] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:01.694398 8315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:01.694653 8315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c466b12d42843219f7d86e5381ecee0, State: Running, Role: LEADER
I20260928 02:55:01.695063 8315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c466b12d42843219f7d86e5381ecee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45391 } }
I20260928 02:55:01.695713 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:01.691415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36125 (local address 127.0.0.1:45391)
0928 02:55:01.691536 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:01.691541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:01.691679 (+ 138us) server_negotiation.cc:407] Connection header received
0928 02:55:01.691829 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:01.691832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:01.691876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:01.691939 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:01.692241 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:01.692834 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:01.693896 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:01.694157 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:01.694232 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:01.695060 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:01.695063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:01.695066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:01.695079 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:01.695110 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:01.695124 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:01.695192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:01.695531 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:55:01.695585 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:55:01.695760 8305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:01.696974 8317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c466b12d42843219f7d86e5381ecee0. Latest consensus state: current_term: 1 leader_uuid: "7c466b12d42843219f7d86e5381ecee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c466b12d42843219f7d86e5381ecee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45391 } } }
I20260928 02:55:01.697104 8317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:01.696975 8316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c466b12d42843219f7d86e5381ecee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c466b12d42843219f7d86e5381ecee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45391 } } }
I20260928 02:55:01.697499 8324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:01.697489 8316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:01.700384 8324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:01.706281 8324 catalog_manager.cc:1347] Generated new cluster ID: 47d3396b223a4abfb99f2e4a4eebf3dc
I20260928 02:55:01.706369 8324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:01.714654 8324 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:01.715965 8324 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:01.723459 8324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c466b12d42843219f7d86e5381ecee0: Generated new TSK 0
I20260928 02:55:01.724390 8324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:01.730594 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8237
2026-09-28T02:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:01 dist-test-slave-5vc0 krb5kdc[8336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:01 dist-test-slave-5vc0 krb5kdc[8336](info): set up 2 sockets
Sep 28 02:55:01 dist-test-slave-5vc0 krb5kdc[8336](info): commencing operation
krb5kdc: starting...
W20260928 02:55:02.680053 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:02 dist-test-slave-5vc0 krb5kdc[8336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:02.803030 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:02.912531 8349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:02.912822 8349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:02.912863 8349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:02.916983 8349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:02.917050 8349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:02.917069 8349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:02.917085 8349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:02.917104 8349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:02.922479 8349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:02.923555 8349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:02.924628 8349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:02.932917 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:02.932917 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:02.932992 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:02.933161 8349 server_base.cc:1034] running on GCE node
I20260928 02:55:02.933790 8349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:02.934942 8349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:02.936115 8349 hybrid_clock.cc:648] HybridClock initialized: now 1790564102936100 us; error 27 us; skew 500 ppm
Sep 28 02:55:02 dist-test-slave-5vc0 krb5kdc[8336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:02.938902 8349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:02.939834 8349 webserver.cc:460] Webserver started at http://127.0.0.1:43289/ using document root <none> and password file <none>
I20260928 02:55:02.940426 8349 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:02.940492 8349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:02.940693 8349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:02.942739 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "34bc98fb7ea341c2b14bf560a153c3fd"
format_stamp: "Formatted at 2026-09-28 02:55:02 on dist-test-slave-5vc0"
server_key: "9f9f62e87e63c2df8a6300454a01fff2"
server_key_iv: "18b68f11382c6bda8dcaa93eeda6955e"
server_key_version: "encryptionkey@0"
I20260928 02:55:02.943286 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "34bc98fb7ea341c2b14bf560a153c3fd"
format_stamp: "Formatted at 2026-09-28 02:55:02 on dist-test-slave-5vc0"
server_key: "9f9f62e87e63c2df8a6300454a01fff2"
server_key_iv: "18b68f11382c6bda8dcaa93eeda6955e"
server_key_version: "encryptionkey@0"
I20260928 02:55:02.946978 8349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:55:02.949704 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:02.951094 8349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:55:02.951241 8349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "34bc98fb7ea341c2b14bf560a153c3fd"
format_stamp: "Formatted at 2026-09-28 02:55:02 on dist-test-slave-5vc0"
server_key: "9f9f62e87e63c2df8a6300454a01fff2"
server_key_iv: "18b68f11382c6bda8dcaa93eeda6955e"
server_key_version: "encryptionkey@0"
I20260928 02:55:02.951355 8349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:02.956128 8349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:02.959546 8349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:02.959738 8349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:02.966523 8349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45591
I20260928 02:55:02.966536 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45591 every 8 connection(s)
I20260928 02:55:02.967448 8349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:02.969326 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8349
I20260928 02:55:02.969462 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:02.969715 20014 external_mini_cluster.cc:1428] Setting key b5b548c25449e8f5a0492a6f602bd5d8
I20260928 02:55:02.970395 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:02.975003 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:02.971000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44847 (local address 127.0.0.1:45591)
0928 02:55:02.971415 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:02.971435 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:02.971626 (+ 191us) server_negotiation.cc:407] Connection header received
0928 02:55:02.972424 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:02.972443 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:02.972734 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:02.973040 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:02.973838 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:02.973843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:02.973855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:02.973865 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:02.973924 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:02.973943 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:02.974070 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:02.974143 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:55:02.974308 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260928 02:55:02.977191 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd: Bootstrap starting.
I20260928 02:55:02.977972 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:02.979606 8417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:02.980525 8417 log.cc:824] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:02.981460 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:02.979876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56525 (local address 127.0.0.1:45591)
0928 02:55:02.979998 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:02.980003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:02.980147 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:55:02.980277 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:02.980281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:02.980324 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:02.980421 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:02.981095 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:02.981099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:02.981101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:02.981114 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:02.981141 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:02.981156 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:02.981254 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:02.981291 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 02:55:02.981333 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:55:02.982838 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd: No bootstrap required, opened a new log
I20260928 02:55:02.986805 8417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260928 02:55:02.987108 8417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:02.987172 8417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bc98fb7ea341c2b14bf560a153c3fd, State: Initialized, Role: FOLLOWER
I20260928 02:55:02.987637 8417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260928 02:55:02.987771 8417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:02.987838 8417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:02.987936 8417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:02.988950 8417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260928 02:55:02.989290 8417 leader_election.cc:302] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34bc98fb7ea341c2b14bf560a153c3fd; no voters:
I20260928 02:55:02.989614 8417 leader_election.cc:288] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:02.989830 8427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:02.990048 8427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [term 1 LEADER]: Becoming Leader. State: Replica: 34bc98fb7ea341c2b14bf560a153c3fd, State: Running, Role: LEADER
I20260928 02:55:02.990392 8427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260928 02:55:02.990808 8417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:02.992084 8429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bc98fb7ea341c2b14bf560a153c3fd. Latest consensus state: current_term: 1 leader_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } } }
I20260928 02:55:02.992127 8428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bc98fb7ea341c2b14bf560a153c3fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } } }
I20260928 02:55:02.992269 8429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:02.992292 8428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:02.992666 8437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:02.995635 8437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:03.001794 8437 catalog_manager.cc:1347] Generated new cluster ID: d9eb3d2a588f4237a47bb8a95a342afa
I20260928 02:55:03.001884 8437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:03.017513 8437 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:03.017908 8437 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:03.032102 8437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34bc98fb7ea341c2b14bf560a153c3fd: Generated new TSK 0
I20260928 02:55:03.033078 8437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:03.045761 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8349
2026-09-28T02:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:03 dist-test-slave-5vc0 krb5kdc[8448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:03 dist-test-slave-5vc0 krb5kdc[8448](info): set up 2 sockets
Sep 28 02:55:03 dist-test-slave-5vc0 krb5kdc[8448](info): commencing operation
krb5kdc: starting...
W20260928 02:55:04.012079 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:04 dist-test-slave-5vc0 krb5kdc[8448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:04.135907 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:04.247143 8461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:04.247457 8461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:04.247501 8461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:04.251675 8461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:04.251762 8461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:04.251781 8461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:04.251797 8461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:04.251812 8461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:04.257355 8461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:04.258440 8461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:04.259467 8461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:04.268201 8469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:04.268121 8466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:04.268139 8461 server_base.cc:1034] running on GCE node
W20260928 02:55:04.268118 8467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:04.269104 8461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:04.270321 8461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:04.271487 8461 hybrid_clock.cc:648] HybridClock initialized: now 1790564104271443 us; error 64 us; skew 500 ppm
Sep 28 02:55:04 dist-test-slave-5vc0 krb5kdc[8448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:04.277397 8461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:04.278363 8461 webserver.cc:460] Webserver started at http://127.0.0.1:36861/ using document root <none> and password file <none>
I20260928 02:55:04.278918 8461 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:04.278981 8461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:04.279177 8461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:04.281277 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "f3ecc4e6125c4dcf85301d40c2830026"
format_stamp: "Formatted at 2026-09-28 02:55:04 on dist-test-slave-5vc0"
server_key: "14a0f0d4bd299914bc41979e0474fa72"
server_key_iv: "3f0108f5c1c8f864109f6dbeee4388f9"
server_key_version: "encryptionkey@0"
I20260928 02:55:04.281831 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "f3ecc4e6125c4dcf85301d40c2830026"
format_stamp: "Formatted at 2026-09-28 02:55:04 on dist-test-slave-5vc0"
server_key: "14a0f0d4bd299914bc41979e0474fa72"
server_key_iv: "3f0108f5c1c8f864109f6dbeee4388f9"
server_key_version: "encryptionkey@0"
I20260928 02:55:04.285584 8461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:55:04.288144 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:04.289479 8461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:55:04.289604 8461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "f3ecc4e6125c4dcf85301d40c2830026"
format_stamp: "Formatted at 2026-09-28 02:55:04 on dist-test-slave-5vc0"
server_key: "14a0f0d4bd299914bc41979e0474fa72"
server_key_iv: "3f0108f5c1c8f864109f6dbeee4388f9"
server_key_version: "encryptionkey@0"
I20260928 02:55:04.289717 8461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:04.303812 8461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:04.306679 8461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:04.306864 8461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:04.313409 8461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46265
I20260928 02:55:04.313426 8528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46265 every 8 connection(s)
I20260928 02:55:04.314356 8461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:04.317134 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:04.322418 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8461
I20260928 02:55:04.322568 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:04.322847 20014 external_mini_cluster.cc:1428] Setting key 3e8adafe9703b33e966bbdb42e5ed058
I20260928 02:55:04.322938 8529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026: Bootstrap starting.
I20260928 02:55:04.325450 8529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:04.326247 8529 log.cc:824] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:04.328320 8529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026: No bootstrap required, opened a new log
Sep 28 02:55:04 dist-test-slave-5vc0 krb5kdc[8448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:55:04.332058 8529 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ecc4e6125c4dcf85301d40c2830026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 02:55:04.332322 8529 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:04.332388 8529 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ecc4e6125c4dcf85301d40c2830026, State: Initialized, Role: FOLLOWER
I20260928 02:55:04.332852 8529 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ecc4e6125c4dcf85301d40c2830026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 02:55:04.333007 8529 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:04.333077 8529 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:04.333184 8529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:04.334038 8529 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ecc4e6125c4dcf85301d40c2830026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 02:55:04.334353 8529 leader_election.cc:302] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3ecc4e6125c4dcf85301d40c2830026; no voters:
I20260928 02:55:04.334640 8529 leader_election.cc:288] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:04.334743 8534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:04.334975 8534 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [term 1 LEADER]: Becoming Leader. State: Replica: f3ecc4e6125c4dcf85301d40c2830026, State: Running, Role: LEADER
I20260928 02:55:04.335304 8534 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ecc4e6125c4dcf85301d40c2830026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 02:55:04.335958 8529 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:04.336959 8536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ecc4e6125c4dcf85301d40c2830026. Latest consensus state: current_term: 1 leader_uuid: "f3ecc4e6125c4dcf85301d40c2830026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ecc4e6125c4dcf85301d40c2830026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260928 02:55:04.337074 8536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:04.337311 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:04.324225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:35845 (local address 127.0.0.1:46265)
0928 02:55:04.324639 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:04.324655 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:04.324691 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:55:04.325333 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:04.325350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:04.325747 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:04.326028 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:04.326890 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:04.327855 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:04.328759 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:04.329235 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:04.329378 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:04.332079 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:04.332083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:04.332097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:04.332124 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:04.333813 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:04.334277 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:04.334282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:04.334289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:04.334410 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:04.334775 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:04.334779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:04.334780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:04.335285 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:04.335486 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:04.335837 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 02:55:04.336277 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260928 02:55:04.337378 8535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3ecc4e6125c4dcf85301d40c2830026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ecc4e6125c4dcf85301d40c2830026" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260928 02:55:04.337468 8535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:04.340519 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 02:55:04.340823 8549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:04.340894 8549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:04.340952 8543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:04.341634 8543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:04.347442 8543 catalog_manager.cc:1347] Generated new cluster ID: 3ecd4417c867454ea90f3cfcbfc0bbe6
I20260928 02:55:04.347590 8543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:04.351370 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:04.342734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:55003 (local address 127.0.0.1:46265)
0928 02:55:04.342842 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:04.342847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:04.343169 (+ 322us) server_negotiation.cc:407] Connection header received
0928 02:55:04.343316 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:04.343319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:04.343370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:04.343451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:04.344089 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:04.344656 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:04.345443 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:04.345755 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:04.345854 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:04.347502 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:04.347506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:04.347508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:04.347535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:04.349205 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:04.349800 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:04.349804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:04.349806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:04.349895 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:04.350327 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:04.350331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:04.350333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:04.350550 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:04.350666 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:04.351092 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 02:55:04.351194 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:55:04.362002 8543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:04.363277 8543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:04.370296 8543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3ecc4e6125c4dcf85301d40c2830026: Generated new TSK 0
I20260928 02:55:04.371054 8543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:04.376065 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8461
W20260928 02:55:04.380282 8446 connection.cc:535] client connection to 127.0.0.1:46265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46265): Connection reset by peer (error 104)
2026-09-28T02:55:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:04 dist-test-slave-5vc0 krb5kdc[8560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:04 dist-test-slave-5vc0 krb5kdc[8560](info): set up 2 sockets
Sep 28 02:55:04 dist-test-slave-5vc0 krb5kdc[8560](info): commencing operation
krb5kdc: starting...
W20260928 02:55:05.340358 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:05 dist-test-slave-5vc0 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:05.461035 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:05.571667 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:05.571974 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:05.572036 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:05.576176 8573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:05.576253 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:05.576282 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:05.576303 8573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:05.576324 8573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:05.581904 8573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:05.582995 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:05.584004 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:05.592516 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:05.592499 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:05.592568 8573 server_base.cc:1034] running on GCE node
W20260928 02:55:05.592499 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:05.593360 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:05.594561 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:05.595798 8573 hybrid_clock.cc:648] HybridClock initialized: now 1790564105595769 us; error 43 us; skew 500 ppm
Sep 28 02:55:05 dist-test-slave-5vc0 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:05.598743 8573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:05.599732 8573 webserver.cc:460] Webserver started at http://127.0.0.1:32869/ using document root <none> and password file <none>
I20260928 02:55:05.600339 8573 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:05.600411 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:05.600621 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:05.602782 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "3e5b6f43af904b06b8e887197236f21a"
format_stamp: "Formatted at 2026-09-28 02:55:05 on dist-test-slave-5vc0"
server_key: "6a544731c22106de3797e5112669865e"
server_key_iv: "0293dcc58fdc3069cd667785b43ae1fb"
server_key_version: "encryptionkey@0"
I20260928 02:55:05.603377 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "3e5b6f43af904b06b8e887197236f21a"
format_stamp: "Formatted at 2026-09-28 02:55:05 on dist-test-slave-5vc0"
server_key: "6a544731c22106de3797e5112669865e"
server_key_iv: "0293dcc58fdc3069cd667785b43ae1fb"
server_key_version: "encryptionkey@0"
I20260928 02:55:05.607380 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 02:55:05.609990 8588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:05.611369 8573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:55:05.611506 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "3e5b6f43af904b06b8e887197236f21a"
format_stamp: "Formatted at 2026-09-28 02:55:05 on dist-test-slave-5vc0"
server_key: "6a544731c22106de3797e5112669865e"
server_key_iv: "0293dcc58fdc3069cd667785b43ae1fb"
server_key_version: "encryptionkey@0"
I20260928 02:55:05.611617 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:05.622541 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:05.625797 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:05.625976 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:05.632637 8573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40509
I20260928 02:55:05.632660 8640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40509 every 8 connection(s)
I20260928 02:55:05.633746 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:05.636663 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:05.637362 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8573
I20260928 02:55:05.637480 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:05.637758 20014 external_mini_cluster.cc:1428] Setting key 407e6d1be80b2cf41dbdcf3b0c43ac74
I20260928 02:55:05.643482 8641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a: Bootstrap starting.
I20260928 02:55:05.646104 8641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:05.647017 8641 log.cc:824] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:05.648102 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:05.639158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:60001 (local address 127.0.0.1:40509)
0928 02:55:05.639607 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:05.639627 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:05.639674 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:55:05.640571 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:05.640595 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:05.641041 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:05.641416 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:05.641823 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:05.643317 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:05.644384 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:05.645090 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:05.645245 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:05.646088 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:05.646095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:05.646112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:05.646641 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:05.646710 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:05.646738 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:05.646870 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:05.647040 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:55:05.647195 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
I20260928 02:55:05.649310 8641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a: No bootstrap required, opened a new log
I20260928 02:55:05.651217 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:05.653146 8641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5b6f43af904b06b8e887197236f21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260928 02:55:05.653426 8641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:05.653527 8641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5b6f43af904b06b8e887197236f21a, State: Initialized, Role: FOLLOWER
I20260928 02:55:05.654000 8641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5b6f43af904b06b8e887197236f21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260928 02:55:05.654170 8641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:05.654235 8641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:05.654331 8641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:05.655328 8641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5b6f43af904b06b8e887197236f21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260928 02:55:05.655652 8641 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e5b6f43af904b06b8e887197236f21a; no voters:
I20260928 02:55:05.655948 8641 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:05.656025 8651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:05.656287 8651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [term 1 LEADER]: Becoming Leader. State: Replica: 3e5b6f43af904b06b8e887197236f21a, State: Running, Role: LEADER
I20260928 02:55:05.656603 8651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5b6f43af904b06b8e887197236f21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260928 02:55:05.657202 8641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:05.657648 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e5b6f43af904b06b8e887197236f21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5b6f43af904b06b8e887197236f21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } } }
I20260928 02:55:05.657764 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:05.658038 8652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5b6f43af904b06b8e887197236f21a. Latest consensus state: current_term: 1 leader_uuid: "3e5b6f43af904b06b8e887197236f21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5b6f43af904b06b8e887197236f21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } } }
I20260928 02:55:05.658180 8652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:05.658468 8656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:05.659030 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:05.653155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:36103 (local address 127.0.0.1:40509)
0928 02:55:05.653263 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:05.653267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:05.653459 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:55:05.653571 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:05.653574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:05.653623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:05.653707 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:05.653965 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:05.654689 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:05.655556 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:05.655852 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:05.655948 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:05.658530 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:05.658534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:05.658537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:05.658551 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:05.658579 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:05.658592 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:05.658674 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:05.658871 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:55:05.658906 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:55:05.662354 8656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:05.668133 8656 catalog_manager.cc:1347] Generated new cluster ID: f7d9438bc1ff4ce6b064fa3da506f247
I20260928 02:55:05.668232 8656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:05.687462 8656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:05.688516 8656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:05.695185 8656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e5b6f43af904b06b8e887197236f21a: Generated new TSK 0
I20260928 02:55:05.696086 8656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:05.708942 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8573
W20260928 02:55:05.713809 8558 connection.cc:535] client connection to 127.0.0.1:40509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40509): Connection reset by peer (error 104)
2026-09-28T02:55:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:05 dist-test-slave-5vc0 krb5kdc[8672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:05 dist-test-slave-5vc0 krb5kdc[8672](info): set up 2 sockets
Sep 28 02:55:05 dist-test-slave-5vc0 krb5kdc[8672](info): commencing operation
krb5kdc: starting...
W20260928 02:55:06.664319 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:06 dist-test-slave-5vc0 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:06.784572 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:06.894022 8685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:06.894315 8685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:06.894356 8685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:06.898370 8685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:06.898437 8685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:06.898456 8685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:06.898473 8685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:06.898486 8685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:06.903890 8685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:06.905020 8685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:06.906069 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:06.914666 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:06.914642 8693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:06.914736 8685 server_base.cc:1034] running on GCE node
W20260928 02:55:06.914642 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:06.915540 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:06.916764 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:06.917953 8685 hybrid_clock.cc:648] HybridClock initialized: now 1790564106917930 us; error 35 us; skew 500 ppm
Sep 28 02:55:06 dist-test-slave-5vc0 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:06.920680 8685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:06.921552 8685 webserver.cc:460] Webserver started at http://127.0.0.1:37923/ using document root <none> and password file <none>
I20260928 02:55:06.922123 8685 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:06.922189 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:06.922394 8685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:06.924505 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "e9869c21c4054738a3e601050de1de47"
format_stamp: "Formatted at 2026-09-28 02:55:06 on dist-test-slave-5vc0"
server_key: "3ed40aeb3ffd74fa65583310aecfc3d4"
server_key_iv: "d02ebd1d0314c2a03e55331add7d498c"
server_key_version: "encryptionkey@0"
I20260928 02:55:06.925050 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "e9869c21c4054738a3e601050de1de47"
format_stamp: "Formatted at 2026-09-28 02:55:06 on dist-test-slave-5vc0"
server_key: "3ed40aeb3ffd74fa65583310aecfc3d4"
server_key_iv: "d02ebd1d0314c2a03e55331add7d498c"
server_key_version: "encryptionkey@0"
I20260928 02:55:06.928747 8685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:06.931295 8700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:06.932637 8685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:06.932763 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "e9869c21c4054738a3e601050de1de47"
format_stamp: "Formatted at 2026-09-28 02:55:06 on dist-test-slave-5vc0"
server_key: "3ed40aeb3ffd74fa65583310aecfc3d4"
server_key_iv: "d02ebd1d0314c2a03e55331add7d498c"
server_key_version: "encryptionkey@0"
I20260928 02:55:06.932874 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:06.937525 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:06.940899 8685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:06.941084 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:06.947611 8685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42039
I20260928 02:55:06.947645 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42039 every 8 connection(s)
I20260928 02:55:06.948599 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:06.950747 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8685
I20260928 02:55:06.950870 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:06.951138 20014 external_mini_cluster.cc:1428] Setting key 14fe20c115d75ed04f72193a84e5e9fe
I20260928 02:55:06.951448 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:06.956486 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:06.952501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:44295 (local address 127.0.0.1:42039)
0928 02:55:06.952908 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:06.952924 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:06.952961 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:55:06.953614 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:06.953630 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:06.953924 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:06.954218 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:06.955081 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:06.955086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:06.955100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:06.955111 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:06.955166 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:06.955188 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:06.955310 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:06.955419 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:55:06.955576 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260928 02:55:06.958417 8753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47: Bootstrap starting.
I20260928 02:55:06.959595 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:06.960783 8753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:06.961540 8753 log.cc:824] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:06.963218 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:06.961643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:49917 (local address 127.0.0.1:42039)
0928 02:55:06.961761 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:06.961765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:06.961936 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:55:06.962055 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:06.962058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:06.962090 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:06.962153 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:06.962846 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:06.962849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:06.962852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:06.962864 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:06.962892 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:06.962902 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:06.962972 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:06.963049 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:55:06.963094 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:55:06.963578 8753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47: No bootstrap required, opened a new log
I20260928 02:55:06.967240 8753 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9869c21c4054738a3e601050de1de47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260928 02:55:06.967496 8753 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:06.967540 8753 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9869c21c4054738a3e601050de1de47, State: Initialized, Role: FOLLOWER
I20260928 02:55:06.967950 8753 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9869c21c4054738a3e601050de1de47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260928 02:55:06.968077 8753 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:06.968125 8753 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:06.968237 8753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:06.969169 8753 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9869c21c4054738a3e601050de1de47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260928 02:55:06.969494 8753 leader_election.cc:302] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9869c21c4054738a3e601050de1de47; no voters:
I20260928 02:55:06.969844 8753 leader_election.cc:288] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:06.969976 8763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:06.970185 8763 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [term 1 LEADER]: Becoming Leader. State: Replica: e9869c21c4054738a3e601050de1de47, State: Running, Role: LEADER
I20260928 02:55:06.970549 8763 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9869c21c4054738a3e601050de1de47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260928 02:55:06.971120 8753 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:06.972344 8764 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9869c21c4054738a3e601050de1de47. Latest consensus state: current_term: 1 leader_uuid: "e9869c21c4054738a3e601050de1de47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9869c21c4054738a3e601050de1de47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } } }
I20260928 02:55:06.972370 8765 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9869c21c4054738a3e601050de1de47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9869c21c4054738a3e601050de1de47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } } }
I20260928 02:55:06.972541 8764 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:06.972544 8765 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47 [sys.catalog]: This master's current role is: LEADER
W20260928 02:55:06.976027 8778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:06.976096 8778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:06.976205 8775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:06.976893 8775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:06.982632 8775 catalog_manager.cc:1347] Generated new cluster ID: 5b6f010dca5d447d8f803695beeaf0b8
I20260928 02:55:06.982715 8775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:07.001518 8775 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:07.001849 8775 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:07.009580 8775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9869c21c4054738a3e601050de1de47: Generated new TSK 0
I20260928 02:55:07.010300 8775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:07.018572 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8685
2026-09-28T02:55:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:07 dist-test-slave-5vc0 krb5kdc[8784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:07 dist-test-slave-5vc0 krb5kdc[8784](info): set up 2 sockets
Sep 28 02:55:07 dist-test-slave-5vc0 krb5kdc[8784](info): commencing operation
krb5kdc: starting...
W20260928 02:55:07.964352 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:08 dist-test-slave-5vc0 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:08.083455 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:08.194733 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:08.195060 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:08.195104 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:08.199293 8797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:08.199371 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:08.199390 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:08.199406 8797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:08.199420 8797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:08.204982 8797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:08.206066 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:08.207069 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:08.215499 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:08.215478 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:08.215539 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:08.215780 8797 server_base.cc:1034] running on GCE node
I20260928 02:55:08.216374 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:08.217500 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:08.218699 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790564108218665 us; error 50 us; skew 500 ppm
Sep 28 02:55:08 dist-test-slave-5vc0 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:08.221602 8797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:08.222514 8797 webserver.cc:460] Webserver started at http://127.0.0.1:36199/ using document root <none> and password file <none>
I20260928 02:55:08.223057 8797 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:08.223125 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:08.223330 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:08.225437 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "ed470f52a8e141a7b6c908c97fb7d968"
format_stamp: "Formatted at 2026-09-28 02:55:08 on dist-test-slave-5vc0"
server_key: "c56f0b0733e69cf42da25ae13046848e"
server_key_iv: "23b979e080d254443de7f920965ef203"
server_key_version: "encryptionkey@0"
I20260928 02:55:08.225987 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "ed470f52a8e141a7b6c908c97fb7d968"
format_stamp: "Formatted at 2026-09-28 02:55:08 on dist-test-slave-5vc0"
server_key: "c56f0b0733e69cf42da25ae13046848e"
server_key_iv: "23b979e080d254443de7f920965ef203"
server_key_version: "encryptionkey@0"
I20260928 02:55:08.229866 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:55:08.232513 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:08.233819 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:55:08.233940 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "ed470f52a8e141a7b6c908c97fb7d968"
format_stamp: "Formatted at 2026-09-28 02:55:08 on dist-test-slave-5vc0"
server_key: "c56f0b0733e69cf42da25ae13046848e"
server_key_iv: "23b979e080d254443de7f920965ef203"
server_key_version: "encryptionkey@0"
I20260928 02:55:08.234045 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:08.247747 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:08.250973 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:08.251159 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:08.258024 8797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34863
I20260928 02:55:08.258039 8864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34863 every 8 connection(s)
I20260928 02:55:08.258972 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:08.259968 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8797
I20260928 02:55:08.260102 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:08.260454 20014 external_mini_cluster.cc:1428] Setting key ef45212d19ccb6de078870cb1a6caea4
I20260928 02:55:08.262362 8865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:08.268034 8865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968: Bootstrap starting.
Sep 28 02:55:08 dist-test-slave-5vc0 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:55:08.270615 8865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:08.271421 8865 log.cc:824] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:08.273803 8865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968: No bootstrap required, opened a new log
I20260928 02:55:08.274659 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:08.261915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:45291 (local address 127.0.0.1:34863)
0928 02:55:08.262397 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:08.262417 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:08.262465 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:55:08.263121 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:08.263137 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:08.263500 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:08.263813 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:08.264647 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:08.265641 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:08.266513 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:08.266991 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:08.267135 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:08.270000 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:08.270006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:08.270021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:08.270052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:08.271736 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:08.272266 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:08.272271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:08.272276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:08.272387 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:08.272699 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:08.272701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:08.272702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:08.273131 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:08.273285 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:08.273611 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:55:08.273892 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":142,"threads_started":1}
I20260928 02:55:08.277891 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:08.277909 8865 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed470f52a8e141a7b6c908c97fb7d968" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260928 02:55:08.278193 8865 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:08.278246 8865 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed470f52a8e141a7b6c908c97fb7d968, State: Initialized, Role: FOLLOWER
I20260928 02:55:08.278738 8865 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed470f52a8e141a7b6c908c97fb7d968" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260928 02:55:08.278913 8865 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:08.278980 8865 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:08.279089 8865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:08.280328 8865 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed470f52a8e141a7b6c908c97fb7d968" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260928 02:55:08.280650 8865 leader_election.cc:302] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed470f52a8e141a7b6c908c97fb7d968; no voters:
I20260928 02:55:08.280910 8865 leader_election.cc:288] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:08.281018 8875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:08.281221 8875 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [term 1 LEADER]: Becoming Leader. State: Replica: ed470f52a8e141a7b6c908c97fb7d968, State: Running, Role: LEADER
I20260928 02:55:08.281582 8875 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed470f52a8e141a7b6c908c97fb7d968" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } }
I20260928 02:55:08.282056 8865 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:08.283231 8877 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed470f52a8e141a7b6c908c97fb7d968. Latest consensus state: current_term: 1 leader_uuid: "ed470f52a8e141a7b6c908c97fb7d968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed470f52a8e141a7b6c908c97fb7d968" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } } }
I20260928 02:55:08.283390 8877 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:08.283215 8876 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed470f52a8e141a7b6c908c97fb7d968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed470f52a8e141a7b6c908c97fb7d968" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34863 } } }
I20260928 02:55:08.283620 8876 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968 [sys.catalog]: This master's current role is: LEADER
W20260928 02:55:08.287309 8890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:08.287421 8890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:08.287521 8886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:08.288300 8886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:08.291893 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:08.280179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:45139 (local address 127.0.0.1:34863)
0928 02:55:08.280294 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:08.280299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:08.284686 (+ 4387us) server_negotiation.cc:407] Connection header received
0928 02:55:08.284839 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:08.284843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:08.284888 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:08.284953 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:08.285641 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:08.286205 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:08.287082 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:08.287324 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:08.287420 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:08.288874 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:08.288877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:08.288879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:08.288902 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:08.290202 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:08.290680 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:08.290683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:08.290684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:08.290761 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:08.291065 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:08.291067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:08.291069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:08.291239 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:08.291397 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:08.291652 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:55:08.291746 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:55:08.294445 8886 catalog_manager.cc:1347] Generated new cluster ID: e0373243c16e4303b3d6be4a2377592e
I20260928 02:55:08.294502 8886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:08.308458 8886 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:08.309530 8886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:08.315448 8886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed470f52a8e141a7b6c908c97fb7d968: Generated new TSK 0
I20260928 02:55:08.316285 8886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:08.323864 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8797
W20260928 02:55:08.328893 8782 connection.cc:535] client connection to 127.0.0.1:34863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34863): Connection reset by peer (error 104)
2026-09-28T02:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:08 dist-test-slave-5vc0 krb5kdc[8896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:08 dist-test-slave-5vc0 krb5kdc[8896](info): set up 2 sockets
Sep 28 02:55:08 dist-test-slave-5vc0 krb5kdc[8896](info): commencing operation
krb5kdc: starting...
W20260928 02:55:09.276329 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:09 dist-test-slave-5vc0 krb5kdc[8896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:09.396957 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:09.506914 8909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:09.507228 8909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:09.507287 8909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:09.511495 8909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:09.511581 8909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:09.511606 8909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:09.511631 8909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:09.511648 8909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:09.517196 8909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:09.518354 8909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:09.519423 8909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:09.527891 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:09.528041 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:09.527891 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:09.528117 8909 server_base.cc:1034] running on GCE node
I20260928 02:55:09.528873 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:09.530058 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:09.531235 8909 hybrid_clock.cc:648] HybridClock initialized: now 1790564109531219 us; error 32 us; skew 500 ppm
Sep 28 02:55:09 dist-test-slave-5vc0 krb5kdc[8896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:09.540210 8909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:09.541285 8909 webserver.cc:460] Webserver started at http://127.0.0.1:35163/ using document root <none> and password file <none>
I20260928 02:55:09.541883 8909 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:09.541957 8909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:09.542178 8909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:09.544533 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "f168be9e23274ed38dc94f838b95c172"
format_stamp: "Formatted at 2026-09-28 02:55:09 on dist-test-slave-5vc0"
server_key: "74091027366068af28d3eee1b9e07038"
server_key_iv: "05c492faaa6f3eab2274fd4285a605c9"
server_key_version: "encryptionkey@0"
I20260928 02:55:09.545109 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "f168be9e23274ed38dc94f838b95c172"
format_stamp: "Formatted at 2026-09-28 02:55:09 on dist-test-slave-5vc0"
server_key: "74091027366068af28d3eee1b9e07038"
server_key_iv: "05c492faaa6f3eab2274fd4285a605c9"
server_key_version: "encryptionkey@0"
I20260928 02:55:09.549270 8909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:55:09.551810 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:09.553385 8909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:09.553525 8909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "f168be9e23274ed38dc94f838b95c172"
format_stamp: "Formatted at 2026-09-28 02:55:09 on dist-test-slave-5vc0"
server_key: "74091027366068af28d3eee1b9e07038"
server_key_iv: "05c492faaa6f3eab2274fd4285a605c9"
server_key_version: "encryptionkey@0"
I20260928 02:55:09.553649 8909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:09.566607 8909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:09.569816 8909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:09.570019 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:09.576923 8909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33143
I20260928 02:55:09.576967 8976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33143 every 8 connection(s)
I20260928 02:55:09.577893 8909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:09.580729 8977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:09.583367 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 8909
I20260928 02:55:09.583501 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:09.583753 20014 external_mini_cluster.cc:1428] Setting key 5e233a0d1c4a428502f9c4cb93ca5a12
I20260928 02:55:09.586699 8977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172: Bootstrap starting.
I20260928 02:55:09.589277 8977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:09.590049 8977 log.cc:824] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:09.592108 8977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172: No bootstrap required, opened a new log
I20260928 02:55:09.592726 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:09.585088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:48557 (local address 127.0.0.1:33143)
0928 02:55:09.585494 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:09.585509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:09.585545 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:55:09.586175 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:09.586193 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:09.586557 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:09.586878 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:09.587207 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:09.588208 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:09.589381 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:09.589885 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:09.590003 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:09.590904 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:09.590909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:09.590922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:09.591309 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:09.591367 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:09.591391 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:09.591509 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:09.591702 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:55:09.591871 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260928 02:55:09.595664 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:09.595863 8977 raft_consensus.cc:348] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f168be9e23274ed38dc94f838b95c172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260928 02:55:09.596098 8977 raft_consensus.cc:374] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:09.596189 8977 raft_consensus.cc:729] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f168be9e23274ed38dc94f838b95c172, State: Initialized, Role: FOLLOWER
I20260928 02:55:09.596638 8977 consensus_queue.cc:260] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f168be9e23274ed38dc94f838b95c172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260928 02:55:09.596764 8977 raft_consensus.cc:388] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:09.596823 8977 raft_consensus.cc:482] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:09.596922 8977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:09.598044 8977 raft_consensus.cc:504] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f168be9e23274ed38dc94f838b95c172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260928 02:55:09.598440 8977 leader_election.cc:302] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f168be9e23274ed38dc94f838b95c172; no voters:
I20260928 02:55:09.598794 8977 leader_election.cc:288] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:09.598894 8987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:09.599051 8987 raft_consensus.cc:686] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [term 1 LEADER]: Becoming Leader. State: Replica: f168be9e23274ed38dc94f838b95c172, State: Running, Role: LEADER
I20260928 02:55:09.599422 8987 consensus_queue.cc:237] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f168be9e23274ed38dc94f838b95c172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260928 02:55:09.600188 8977 sys_catalog.cc:564] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:09.601740 8989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f168be9e23274ed38dc94f838b95c172. Latest consensus state: current_term: 1 leader_uuid: "f168be9e23274ed38dc94f838b95c172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f168be9e23274ed38dc94f838b95c172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } } }
I20260928 02:55:09.601871 8989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:09.602216 9001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:09.602500 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:09.597733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:34175 (local address 127.0.0.1:33143)
0928 02:55:09.597867 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:09.597873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:09.598121 (+ 248us) server_negotiation.cc:407] Connection header received
0928 02:55:09.598258 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:09.598261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:09.598315 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:09.598395 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:09.598719 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:09.599429 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:09.600341 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:09.600597 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:09.600685 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:09.601782 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:09.601786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:09.601789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:09.601803 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:09.601830 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:09.601843 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:09.601938 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:09.602332 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:55:09.602368 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:55:09.606302 9001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:09.602553 8988 sys_catalog.cc:455] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f168be9e23274ed38dc94f838b95c172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f168be9e23274ed38dc94f838b95c172" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } } }
I20260928 02:55:09.606457 8988 sys_catalog.cc:458] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:09.612444 9001 catalog_manager.cc:1347] Generated new cluster ID: 3fff4f69e401435f908e85fd14bd925a
I20260928 02:55:09.612530 9001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:09.625108 9001 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:09.626389 9001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:09.636225 9001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f168be9e23274ed38dc94f838b95c172: Generated new TSK 0
I20260928 02:55:09.636941 9001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:09.644312 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 8909
W20260928 02:55:09.648918 8894 connection.cc:535] client connection to 127.0.0.1:33143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33143): Connection reset by peer (error 104)
2026-09-28T02:55:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:09 dist-test-slave-5vc0 krb5kdc[9008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:09 dist-test-slave-5vc0 krb5kdc[9008](info): set up 2 sockets
Sep 28 02:55:09 dist-test-slave-5vc0 krb5kdc[9008](info): commencing operation
krb5kdc: starting...
W20260928 02:55:10.596359 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:10 dist-test-slave-5vc0 krb5kdc[9008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:55:10.715831 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:10.826529 9021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:10.826830 9021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:10.826874 9021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:10.830892 9021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:10.830960 9021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:10.830979 9021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:10.830994 9021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:10.831008 9021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:10.836483 9021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:10.837587 9021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:10.838644 9021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:10.847049 9026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:10.847051 9027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:10.847049 9029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:10.847445 9021 server_base.cc:1034] running on GCE node
I20260928 02:55:10.848003 9021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:10.849172 9021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:10.850354 9021 hybrid_clock.cc:648] HybridClock initialized: now 1790564110850319 us; error 32 us; skew 500 ppm
Sep 28 02:55:10 dist-test-slave-5vc0 krb5kdc[9008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:10.853156 9021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:55:10.854048 9021 webserver.cc:460] Webserver started at http://127.0.0.1:43123/ using document root <none> and password file <none>
I20260928 02:55:10.854588 9021 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:10.854655 9021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:10.854862 9021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:10.857023 9021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "23bc361f4c574d668a73cdce071296d4"
format_stamp: "Formatted at 2026-09-28 02:55:10 on dist-test-slave-5vc0"
server_key: "4369b9d8a1bcbb5fdd42b33c366892ef"
server_key_iv: "2ef7f5fb77c7cb6aa583f8b98ca1b2ba"
server_key_version: "encryptionkey@0"
I20260928 02:55:10.857568 9021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "23bc361f4c574d668a73cdce071296d4"
format_stamp: "Formatted at 2026-09-28 02:55:10 on dist-test-slave-5vc0"
server_key: "4369b9d8a1bcbb5fdd42b33c366892ef"
server_key_iv: "2ef7f5fb77c7cb6aa583f8b98ca1b2ba"
server_key_version: "encryptionkey@0"
I20260928 02:55:10.861496 9021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 02:55:10.863979 9036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:10.865350 9021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:55:10.865474 9021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "23bc361f4c574d668a73cdce071296d4"
format_stamp: "Formatted at 2026-09-28 02:55:10 on dist-test-slave-5vc0"
server_key: "4369b9d8a1bcbb5fdd42b33c366892ef"
server_key_iv: "2ef7f5fb77c7cb6aa583f8b98ca1b2ba"
server_key_version: "encryptionkey@0"
I20260928 02:55:10.865578 9021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:10.870381 9021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:10.873487 9021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:10.873665 9021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:10.880306 9021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42155
I20260928 02:55:10.880319 9088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42155 every 8 connection(s)
I20260928 02:55:10.881235 9021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:10.881979 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9021
I20260928 02:55:10.882120 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:10.882465 20014 external_mini_cluster.cc:1428] Setting key 694393f28b969175f76899161c42b8c5
I20260928 02:55:10.884634 9089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:10.888116 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:10.883888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:47109 (local address 127.0.0.1:42155)
0928 02:55:10.884367 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:10.884383 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:10.884597 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:55:10.885437 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:10.885460 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:10.885760 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:10.886051 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:10.886890 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:10.886894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:10.886906 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:10.886918 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:10.886971 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:10.886991 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:10.887100 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:10.887208 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:55:10.887362 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260928 02:55:10.890600 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4: Bootstrap starting.
I20260928 02:55:10.890710 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:10.893339 9089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:10.894182 9089 log.cc:824] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:10.894464 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:10.892925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:41193 (local address 127.0.0.1:42155)
0928 02:55:10.893045 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:10.893049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:10.893196 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:55:10.893328 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:10.893331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:10.893361 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:10.893432 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:10.894094 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:10.894097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:10.894099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:10.894108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:10.894131 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:10.894141 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:10.894205 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:10.894305 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:55:10.894344 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:55:10.896250 9089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4: No bootstrap required, opened a new log
I20260928 02:55:10.899768 9089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bc361f4c574d668a73cdce071296d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260928 02:55:10.900053 9089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:10.900118 9089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bc361f4c574d668a73cdce071296d4, State: Initialized, Role: FOLLOWER
I20260928 02:55:10.900628 9089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bc361f4c574d668a73cdce071296d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260928 02:55:10.900758 9089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:10.900820 9089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:10.900913 9089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:10.901822 9089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bc361f4c574d668a73cdce071296d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260928 02:55:10.902155 9089 leader_election.cc:302] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23bc361f4c574d668a73cdce071296d4; no voters:
I20260928 02:55:10.902455 9089 leader_election.cc:288] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:10.902560 9099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:10.902832 9099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [term 1 LEADER]: Becoming Leader. State: Replica: 23bc361f4c574d668a73cdce071296d4, State: Running, Role: LEADER
I20260928 02:55:10.903108 9099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bc361f4c574d668a73cdce071296d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } }
I20260928 02:55:10.903644 9089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:10.904667 9100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23bc361f4c574d668a73cdce071296d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bc361f4c574d668a73cdce071296d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } } }
I20260928 02:55:10.904793 9100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:10.905068 9101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bc361f4c574d668a73cdce071296d4. Latest consensus state: current_term: 1 leader_uuid: "23bc361f4c574d668a73cdce071296d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bc361f4c574d668a73cdce071296d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42155 } } }
I20260928 02:55:10.905158 9101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:10.905233 9108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:10.908381 9108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:10.914167 9108 catalog_manager.cc:1347] Generated new cluster ID: 9ae6712e2f754d5e818c1f53e5474f5c
I20260928 02:55:10.914234 9108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:10.921802 9108 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:10.922115 9108 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:10.930194 9108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23bc361f4c574d668a73cdce071296d4: Generated new TSK 0
I20260928 02:55:10.931030 9108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:10.943209 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9021
2026-09-28T02:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:10 dist-test-slave-5vc0 krb5kdc[9120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:10 dist-test-slave-5vc0 krb5kdc[9120](info): set up 2 sockets
Sep 28 02:55:10 dist-test-slave-5vc0 krb5kdc[9120](info): commencing operation
krb5kdc: starting...
W20260928 02:55:11.888321 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:11 dist-test-slave-5vc0 krb5kdc[9120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:12.010782 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:44783
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:44783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:12.120268 9133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:12.120541 9133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:12.120582 9133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:12.124608 9133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:12.124675 9133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:12.124693 9133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:12.124708 9133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:12.124722 9133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:12.130127 9133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:44783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:44783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:12.131186 9133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:12.132210 9133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:12.140455 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:12.140441 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:12.140439 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:12.140826 9133 server_base.cc:1034] running on GCE node
I20260928 02:55:12.141382 9133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:12.142453 9133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:12.143621 9133 hybrid_clock.cc:648] HybridClock initialized: now 1790564112143602 us; error 32 us; skew 500 ppm
Sep 28 02:55:12 dist-test-slave-5vc0 krb5kdc[9120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564112, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:12.146812 9133 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:12.147907 9133 webserver.cc:460] Webserver started at http://127.19.139.190:38575/ using document root <none> and password file <none>
I20260928 02:55:12.148484 9133 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:12.148547 9133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:12.148720 9133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:12.150787 9133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "2f5d6a10d247462986fde78ce61c89e8"
format_stamp: "Formatted at 2026-09-28 02:55:12 on dist-test-slave-5vc0"
server_key: "f05bfd352f350c2bdc794626d1fe17a2"
server_key_iv: "01f97b0cde9370dd9714a685304ee478"
server_key_version: "encryptionkey@0"
I20260928 02:55:12.151340 9133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "2f5d6a10d247462986fde78ce61c89e8"
format_stamp: "Formatted at 2026-09-28 02:55:12 on dist-test-slave-5vc0"
server_key: "f05bfd352f350c2bdc794626d1fe17a2"
server_key_iv: "01f97b0cde9370dd9714a685304ee478"
server_key_version: "encryptionkey@0"
I20260928 02:55:12.155160 9133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 02:55:12.157683 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:12.159162 9133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 02:55:12.159283 9133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "2f5d6a10d247462986fde78ce61c89e8"
format_stamp: "Formatted at 2026-09-28 02:55:12 on dist-test-slave-5vc0"
server_key: "f05bfd352f350c2bdc794626d1fe17a2"
server_key_iv: "01f97b0cde9370dd9714a685304ee478"
server_key_version: "encryptionkey@0"
I20260928 02:55:12.159382 9133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:12.182302 9133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:12.185679 9133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:12.185858 9133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:12.192654 9133 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:44783
I20260928 02:55:12.192683 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:44783 every 8 connection(s)
I20260928 02:55:12.193674 9133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:12.196447 9201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:12.197245 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9133
I20260928 02:55:12.197386 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:12.197645 20014 external_mini_cluster.cc:1428] Setting key da71d71f051f2601f6536c0cfbd43d88
I20260928 02:55:12.203051 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8: Bootstrap starting.
I20260928 02:55:12.205681 9201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:12.206408 9201 log.cc:824] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:55:12 dist-test-slave-5vc0 krb5kdc[9120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:55:12.208570 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8: No bootstrap required, opened a new log
I20260928 02:55:12.212121 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:12.198998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46431 (local address 127.19.139.190:44783)
0928 02:55:12.199434 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:12.199453 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:12.199499 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:55:12.200321 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:12.200355 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:12.200786 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:12.201153 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:12.201824 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:12.203296 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:12.204292 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:12.204984 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:12.205145 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:12.207725 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:12.207731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:12.207745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:12.207772 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:12.209347 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:12.209903 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:12.209908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:12.209912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:12.210021 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:12.210320 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:12.210322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:12.210323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:12.210680 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:12.210826 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:12.211127 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:55:12.211337 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260928 02:55:12.212527 9201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5d6a10d247462986fde78ce61c89e8" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44783 } }
I20260928 02:55:12.212802 9201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:12.212859 9201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5d6a10d247462986fde78ce61c89e8, State: Initialized, Role: FOLLOWER
I20260928 02:55:12.213306 9201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5d6a10d247462986fde78ce61c89e8" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44783 } }
I20260928 02:55:12.213455 9201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:12.213519 9201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:12.213617 9201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:12.214668 9201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5d6a10d247462986fde78ce61c89e8" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44783 } }
I20260928 02:55:12.214999 9201 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5d6a10d247462986fde78ce61c89e8; no voters:
I20260928 02:55:12.215173 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:12.215346 9201 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:12.215430 9206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:12.215618 9206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [term 1 LEADER]: Becoming Leader. State: Replica: 2f5d6a10d247462986fde78ce61c89e8, State: Running, Role: LEADER
I20260928 02:55:12.216001 9206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5d6a10d247462986fde78ce61c89e8" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44783 } }
I20260928 02:55:12.216589 9201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:12.217667 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5d6a10d247462986fde78ce61c89e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5d6a10d247462986fde78ce61c89e8" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44783 } } }
I20260928 02:55:12.217790 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:12.218160 9207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5d6a10d247462986fde78ce61c89e8. Latest consensus state: current_term: 1 leader_uuid: "2f5d6a10d247462986fde78ce61c89e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5d6a10d247462986fde78ce61c89e8" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 44783 } } }
I20260928 02:55:12.218317 9207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:12.219473 9219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:12.222582 9219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:12.227065 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:12.217574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47099 (local address 127.19.139.190:44783)
0928 02:55:12.217681 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:12.217685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:12.218261 (+ 576us) server_negotiation.cc:407] Connection header received
0928 02:55:12.218449 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:12.218453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:12.218501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:12.218553 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:12.219464 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:12.220077 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:12.221181 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:12.221542 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:12.221652 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:12.223282 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:12.223286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:12.223289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:12.223317 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:12.224998 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:12.225579 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:12.225583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:12.225586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:12.225682 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:12.226046 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:12.226049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:12.226051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:12.226276 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:12.226433 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:12.226809 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 02:55:12.226915 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:55:12.229103 9219 catalog_manager.cc:1347] Generated new cluster ID: 6cff65edd79a46f68c7b76a7897491e9
I20260928 02:55:12.229171 9219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:12.246223 9219 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:12.247198 9219 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:12.257915 9219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5d6a10d247462986fde78ce61c89e8: Generated new TSK 0
I20260928 02:55:12.258651 9219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:12.274184 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9133
W20260928 02:55:12.278985 9118 connection.cc:535] client connection to 127.19.139.190:44783 recv error: Network error: failed to read from TLS socket (remote: 127.19.139.190:44783): Connection reset by peer (error 104)
2026-09-28T02:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:12 dist-test-slave-5vc0 krb5kdc[9232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:12 dist-test-slave-5vc0 krb5kdc[9232](info): set up 2 sockets
Sep 28 02:55:12 dist-test-slave-5vc0 krb5kdc[9232](info): commencing operation
krb5kdc: starting...
W20260928 02:55:13.228291 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:13 dist-test-slave-5vc0 krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:13.348714 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45449
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:13.459384 9245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:13.459681 9245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:13.459729 9245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:13.463785 9245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:13.463861 9245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:13.463881 9245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:13.463896 9245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:13.463909 9245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:13.469501 9245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:13.470649 9245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:13.471796 9245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:55:13.480496 9245 server_base.cc:1034] running on GCE node
W20260928 02:55:13.480432 9251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:13.480443 9250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:13.480439 9253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:13.481318 9245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:13.482486 9245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:13.483645 9245 hybrid_clock.cc:648] HybridClock initialized: now 1790564113483625 us; error 35 us; skew 500 ppm
Sep 28 02:55:13 dist-test-slave-5vc0 krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564113, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:13.489925 9245 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:13.490908 9245 webserver.cc:460] Webserver started at http://127.19.139.190:35753/ using document root <none> and password file <none>
I20260928 02:55:13.491503 9245 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:13.491572 9245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:13.491777 9245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:13.493966 9245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83"
format_stamp: "Formatted at 2026-09-28 02:55:13 on dist-test-slave-5vc0"
server_key: "56e341f96fbb3732373d4e0f4fc71164"
server_key_iv: "c885f24aaeb186103a7d95c05ffa2013"
server_key_version: "encryptionkey@0"
I20260928 02:55:13.494527 9245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83"
format_stamp: "Formatted at 2026-09-28 02:55:13 on dist-test-slave-5vc0"
server_key: "56e341f96fbb3732373d4e0f4fc71164"
server_key_iv: "c885f24aaeb186103a7d95c05ffa2013"
server_key_version: "encryptionkey@0"
I20260928 02:55:13.498417 9245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 02:55:13.500878 9260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:13.502164 9245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:55:13.502282 9245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83"
format_stamp: "Formatted at 2026-09-28 02:55:13 on dist-test-slave-5vc0"
server_key: "56e341f96fbb3732373d4e0f4fc71164"
server_key_iv: "c885f24aaeb186103a7d95c05ffa2013"
server_key_version: "encryptionkey@0"
I20260928 02:55:13.502384 9245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:13.520711 9245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:13.524250 9245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:13.524442 9245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:13.531443 9245 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:45449
I20260928 02:55:13.531510 9312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:45449 every 8 connection(s)
I20260928 02:55:13.532589 9245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:13.535236 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9245
I20260928 02:55:13.535395 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:13.535617 9313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:13.535727 20014 external_mini_cluster.cc:1428] Setting key 7cc96bd345911d181d17642565ed3b4e
I20260928 02:55:13.542663 9313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83: Bootstrap starting.
I20260928 02:55:13.545200 9313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:13.545540 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:13.537204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36597 (local address 127.19.139.190:45449)
0928 02:55:13.537708 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:13.537724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:13.537912 (+ 188us) server_negotiation.cc:407] Connection header received
0928 02:55:13.538906 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:13.538924 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:13.539273 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:13.539565 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:13.539890 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:13.540932 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:13.542114 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:13.542646 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:13.542768 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:13.543726 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:13.543730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:13.543743 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:13.544206 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:13.544268 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:13.544293 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:13.544395 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:13.544589 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:55:13.544724 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":155,"threads_started":1}
I20260928 02:55:13.545979 9313 log.cc:824] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:13.548026 9313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83: No bootstrap required, opened a new log
I20260928 02:55:13.548398 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:13.552050 9313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45449 } }
I20260928 02:55:13.552371 9313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:13.552433 9313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dcf1b0b3cfd47eca88ed44e9c9ede83, State: Initialized, Role: FOLLOWER
I20260928 02:55:13.552911 9313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45449 } }
I20260928 02:55:13.553081 9313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:13.553184 9313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:13.553295 9313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:13.554456 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:13.550267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51921 (local address 127.19.139.190:45449)
0928 02:55:13.550392 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:13.550396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:13.550517 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:55:13.550607 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:13.550611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:13.550658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:13.550741 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:13.550958 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:13.551707 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:13.552633 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:13.552986 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:13.553089 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:13.554023 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:13.554026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:13.554029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:13.554043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:13.554074 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:13.554089 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:13.554170 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:13.554285 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:55:13.554324 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:55:13.554530 9313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45449 } }
I20260928 02:55:13.554911 9313 leader_election.cc:302] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dcf1b0b3cfd47eca88ed44e9c9ede83; no voters:
I20260928 02:55:13.555244 9313 leader_election.cc:288] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:13.555348 9323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:13.555605 9323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [term 1 LEADER]: Becoming Leader. State: Replica: 2dcf1b0b3cfd47eca88ed44e9c9ede83, State: Running, Role: LEADER
I20260928 02:55:13.555972 9323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45449 } }
I20260928 02:55:13.556550 9313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:13.557588 9324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45449 } } }
I20260928 02:55:13.557760 9324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:13.558039 9324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dcf1b0b3cfd47eca88ed44e9c9ede83. Latest consensus state: current_term: 1 leader_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dcf1b0b3cfd47eca88ed44e9c9ede83" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45449 } } }
I20260928 02:55:13.558183 9324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:13.558295 9332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:13.561432 9332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:13.567210 9332 catalog_manager.cc:1347] Generated new cluster ID: b02b625db0a34d03bfda90dcffd6e6d7
I20260928 02:55:13.567297 9332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:13.582710 9332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:13.583668 9332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:13.592056 9332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dcf1b0b3cfd47eca88ed44e9c9ede83: Generated new TSK 0
I20260928 02:55:13.592974 9332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:13.608093 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9245
W20260928 02:55:13.612617 9230 connection.cc:535] client connection to 127.19.139.190:45449 recv error: Network error: failed to read from TLS socket (remote: 127.19.139.190:45449): Connection reset by peer (error 104)
2026-09-28T02:55:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:13 dist-test-slave-5vc0 krb5kdc[9344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:13 dist-test-slave-5vc0 krb5kdc[9344](info): set up 2 sockets
Sep 28 02:55:13 dist-test-slave-5vc0 krb5kdc[9344](info): commencing operation
krb5kdc: starting...
W20260928 02:55:14.572235 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:14 dist-test-slave-5vc0 krb5kdc[9344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:14.694918 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:43623
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:14.805734 9357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:14.806020 9357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:14.806061 9357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:14.810074 9357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:14.810149 9357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:14.810168 9357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:14.810182 9357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:14.810196 9357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:14.815594 9357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:43623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:14.816789 9357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:14.817812 9357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:14.826453 9363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:14.826432 9365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:14.826593 9357 server_base.cc:1034] running on GCE node
W20260928 02:55:14.826436 9362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:14.827399 9357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:14.828569 9357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:14.829725 9357 hybrid_clock.cc:648] HybridClock initialized: now 1790564114829701 us; error 47 us; skew 500 ppm
Sep 28 02:55:14 dist-test-slave-5vc0 krb5kdc[9344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564114, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:14.835420 9357 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:14.836498 9357 webserver.cc:460] Webserver started at http://127.19.139.190:36927/ using document root <none> and password file <none>
I20260928 02:55:14.837100 9357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:14.837172 9357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:14.837366 9357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:14.839484 9357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "1a2accb5673f427695f10be419a14d6b"
format_stamp: "Formatted at 2026-09-28 02:55:14 on dist-test-slave-5vc0"
server_key: "4701411d825db0eeb5bc439d5e50bbae"
server_key_iv: "04af43f8f3d368fab32a63d1dfeb2544"
server_key_version: "encryptionkey@0"
I20260928 02:55:14.840050 9357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "1a2accb5673f427695f10be419a14d6b"
format_stamp: "Formatted at 2026-09-28 02:55:14 on dist-test-slave-5vc0"
server_key: "4701411d825db0eeb5bc439d5e50bbae"
server_key_iv: "04af43f8f3d368fab32a63d1dfeb2544"
server_key_version: "encryptionkey@0"
I20260928 02:55:14.843856 9357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:55:14.846535 9372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:14.847821 9357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:55:14.847970 9357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "1a2accb5673f427695f10be419a14d6b"
format_stamp: "Formatted at 2026-09-28 02:55:14 on dist-test-slave-5vc0"
server_key: "4701411d825db0eeb5bc439d5e50bbae"
server_key_iv: "04af43f8f3d368fab32a63d1dfeb2544"
server_key_version: "encryptionkey@0"
I20260928 02:55:14.848073 9357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:14.852700 9357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:14.855726 9357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:14.855916 9357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:14.862855 9357 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:43623
I20260928 02:55:14.862900 9424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:43623 every 8 connection(s)
I20260928 02:55:14.863839 9357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:14.866712 9425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:14.871166 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9357
I20260928 02:55:14.871289 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:14.871553 20014 external_mini_cluster.cc:1428] Setting key 6d2b6b37a8779ac49f9669b7747a9184
I20260928 02:55:14.872717 9425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b: Bootstrap starting.
I20260928 02:55:14.874991 9425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:14.875890 9425 log.cc:824] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:14.876924 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:14.873014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35053 (local address 127.19.139.190:43623)
0928 02:55:14.873421 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:14.873437 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:14.873470 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:55:14.874129 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:14.874147 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:14.874442 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:14.874733 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:14.875545 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:14.875549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:14.875561 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:14.875571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:14.875627 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:14.875646 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:14.875729 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:14.875851 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:55:14.876034 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260928 02:55:14.877945 9425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b: No bootstrap required, opened a new log
I20260928 02:55:14.879472 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:14.882108 9425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2accb5673f427695f10be419a14d6b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43623 } }
I20260928 02:55:14.882416 9425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:14.882476 9425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2accb5673f427695f10be419a14d6b, State: Initialized, Role: FOLLOWER
I20260928 02:55:14.882929 9425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2accb5673f427695f10be419a14d6b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43623 } }
I20260928 02:55:14.883087 9425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:14.883160 9425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:14.883275 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:14.881910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39801 (local address 127.19.139.190:43623)
0928 02:55:14.882016 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:14.882019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:14.882031 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:55:14.882119 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:14.882122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:14.882149 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:14.882208 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:14.882886 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:14.882889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:14.882891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:14.882900 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:14.882926 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:14.882936 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:14.883000 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:14.883096 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:55:14.883133 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:55:14.883313 9425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:14.884280 9425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2accb5673f427695f10be419a14d6b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43623 } }
I20260928 02:55:14.884605 9425 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2accb5673f427695f10be419a14d6b; no voters:
I20260928 02:55:14.884894 9425 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:14.885025 9435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:14.885207 9435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [term 1 LEADER]: Becoming Leader. State: Replica: 1a2accb5673f427695f10be419a14d6b, State: Running, Role: LEADER
I20260928 02:55:14.885526 9435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2accb5673f427695f10be419a14d6b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43623 } }
I20260928 02:55:14.886031 9425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:14.887091 9437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2accb5673f427695f10be419a14d6b. Latest consensus state: current_term: 1 leader_uuid: "1a2accb5673f427695f10be419a14d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2accb5673f427695f10be419a14d6b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43623 } } }
I20260928 02:55:14.887094 9436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2accb5673f427695f10be419a14d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2accb5673f427695f10be419a14d6b" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 43623 } } }
I20260928 02:55:14.887277 9437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:14.887291 9436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:14.887672 9444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:14.890787 9444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:14.896117 9444 catalog_manager.cc:1347] Generated new cluster ID: 1de9b472ddd9450d9cc960559b8e8f44
I20260928 02:55:14.896219 9444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:14.910301 9444 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:14.910629 9444 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:14.914902 9444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2accb5673f427695f10be419a14d6b: Generated new TSK 0
I20260928 02:55:14.915581 9444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:14.921506 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9357
2026-09-28T02:55:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:14 dist-test-slave-5vc0 krb5kdc[9456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:14 dist-test-slave-5vc0 krb5kdc[9456](info): set up 2 sockets
Sep 28 02:55:14 dist-test-slave-5vc0 krb5kdc[9456](info): commencing operation
krb5kdc: starting...
W20260928 02:55:15.868245 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:15 dist-test-slave-5vc0 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:15.985190 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40525
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:16.097069 9469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:16.097378 9469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:16.097445 9469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:16.101969 9469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:16.102047 9469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:16.102077 9469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:16.102099 9469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:16.102128 9469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:16.107671 9469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:40525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:16.108844 9469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:16.109920 9469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:16.118217 9474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:16.118299 9477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:16.118311 9475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:16.118450 9469 server_base.cc:1034] running on GCE node
I20260928 02:55:16.119107 9469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:16.120247 9469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:16.121409 9469 hybrid_clock.cc:648] HybridClock initialized: now 1790564116121395 us; error 29 us; skew 500 ppm
Sep 28 02:55:16 dist-test-slave-5vc0 krb5kdc[9456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564116, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:16.124043 9469 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:16.124954 9469 webserver.cc:460] Webserver started at http://127.19.139.190:37151/ using document root <none> and password file <none>
I20260928 02:55:16.125497 9469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:16.125562 9469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:16.125759 9469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:16.127822 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "7044cb78edb340c59f48d9a60be13295"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-5vc0"
server_key: "5b4b7bc7b22f0929093078c6dae592dc"
server_key_iv: "54ff12f4147f44a66ebe18c461c130e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:16.128401 9469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "7044cb78edb340c59f48d9a60be13295"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-5vc0"
server_key: "5b4b7bc7b22f0929093078c6dae592dc"
server_key_iv: "54ff12f4147f44a66ebe18c461c130e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:16.132102 9469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:55:16.134527 9484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:16.135741 9469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:55:16.135865 9469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "7044cb78edb340c59f48d9a60be13295"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-5vc0"
server_key: "5b4b7bc7b22f0929093078c6dae592dc"
server_key_iv: "54ff12f4147f44a66ebe18c461c130e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:16.135964 9469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:16.147032 9469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:16.150023 9469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:16.150206 9469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:16.156854 9469 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:40525
I20260928 02:55:16.156888 9536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:40525 every 8 connection(s)
I20260928 02:55:16.157799 9469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:16.160892 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:16.161341 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9469
I20260928 02:55:16.161489 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:16.161784 20014 external_mini_cluster.cc:1428] Setting key 716151ed98052303231a52ecf0cfb8f6
I20260928 02:55:16.167621 9537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295: Bootstrap starting.
I20260928 02:55:16.170470 9537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:16.171437 9537 log.cc:824] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:55:16 dist-test-slave-5vc0 krb5kdc[9456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:55:16.173782 9537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295: No bootstrap required, opened a new log
I20260928 02:55:16.177202 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:16.163538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35983 (local address 127.19.139.190:40525)
0928 02:55:16.163960 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:16.163979 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:16.164014 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:55:16.164714 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:16.164732 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:16.165095 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:16.165391 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:16.166325 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:16.167821 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:16.169080 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:16.169609 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:16.169754 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:16.172644 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:16.172651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:16.172665 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:16.172695 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:16.174327 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:16.174762 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:16.174768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:16.174773 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:16.174893 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:16.175155 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:16.175158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:16.175159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:16.175578 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:16.175759 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:16.176047 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:55:16.176412 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
I20260928 02:55:16.177470 9537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7044cb78edb340c59f48d9a60be13295" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40525 } }
I20260928 02:55:16.177706 9537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:16.177762 9537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7044cb78edb340c59f48d9a60be13295, State: Initialized, Role: FOLLOWER
I20260928 02:55:16.178223 9537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7044cb78edb340c59f48d9a60be13295" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40525 } }
I20260928 02:55:16.178393 9537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:16.178457 9537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:16.178551 9537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:16.179711 9537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7044cb78edb340c59f48d9a60be13295" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40525 } }
I20260928 02:55:16.180096 9537 leader_election.cc:302] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7044cb78edb340c59f48d9a60be13295; no voters:
I20260928 02:55:16.180302 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:16.180472 9537 leader_election.cc:288] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:16.180579 9542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:16.180749 9542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [term 1 LEADER]: Becoming Leader. State: Replica: 7044cb78edb340c59f48d9a60be13295, State: Running, Role: LEADER
I20260928 02:55:16.181118 9542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7044cb78edb340c59f48d9a60be13295" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40525 } }
I20260928 02:55:16.181824 9537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:16.182335 9542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7044cb78edb340c59f48d9a60be13295. Latest consensus state: current_term: 1 leader_uuid: "7044cb78edb340c59f48d9a60be13295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7044cb78edb340c59f48d9a60be13295" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40525 } } }
I20260928 02:55:16.182520 9542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:16.182729 9543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7044cb78edb340c59f48d9a60be13295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7044cb78edb340c59f48d9a60be13295" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 40525 } } }
I20260928 02:55:16.182830 9543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:16.182898 9553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:16.186286 9553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:16.192051 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:16.183002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55577 (local address 127.19.139.190:40525)
0928 02:55:16.183105 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:16.183109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:16.184043 (+ 934us) server_negotiation.cc:407] Connection header received
0928 02:55:16.184196 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:16.184199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:16.184247 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:16.184310 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:16.185132 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:16.185677 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:16.186769 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:16.187057 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:16.187115 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:16.188703 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:16.188707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:16.188711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:16.188739 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:16.190379 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:16.190794 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:16.190798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:16.190800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:16.190903 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:16.191181 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:16.191185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:16.191187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:16.191433 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:16.191560 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:16.191748 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:55:16.191871 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:55:16.197387 9553 catalog_manager.cc:1347] Generated new cluster ID: 797e6975aadd49c1bfe58751925ce26a
I20260928 02:55:16.197562 9553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:16.230448 9553 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:16.231413 9553 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:16.237983 9553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7044cb78edb340c59f48d9a60be13295: Generated new TSK 0
I20260928 02:55:16.238699 9553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:16.253656 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9469
2026-09-28T02:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:16 dist-test-slave-5vc0 krb5kdc[9568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:16 dist-test-slave-5vc0 krb5kdc[9568](info): set up 2 sockets
Sep 28 02:55:16 dist-test-slave-5vc0 krb5kdc[9568](info): commencing operation
krb5kdc: starting...
W20260928 02:55:17.204131 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:17 dist-test-slave-5vc0 krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:17.323900 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35981
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:17.436455 9581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:17.436762 9581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:17.436822 9581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:17.440940 9581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:17.441016 9581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:17.441042 9581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:17.441064 9581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:17.441095 9581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:17.446593 9581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:35981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:17.447726 9581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:17.448823 9581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:17.457142 9589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:17.457130 9587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:17.457159 9586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:17.457520 9581 server_base.cc:1034] running on GCE node
I20260928 02:55:17.458137 9581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:17.459337 9581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:17.460515 9581 hybrid_clock.cc:648] HybridClock initialized: now 1790564117460498 us; error 31 us; skew 500 ppm
Sep 28 02:55:17 dist-test-slave-5vc0 krb5kdc[9568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:17.463366 9581 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:17.464383 9581 webserver.cc:460] Webserver started at http://127.19.139.190:43855/ using document root <none> and password file <none>
I20260928 02:55:17.464958 9581 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:17.465031 9581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:17.465243 9581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:17.467682 9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-5vc0"
server_key: "1d75afc02c3cde4264616dcd7d06dd8a"
server_key_iv: "b65027bb0a7b04ece16dd30cbc981503"
server_key_version: "encryptionkey@0"
I20260928 02:55:17.468318 9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-5vc0"
server_key: "1d75afc02c3cde4264616dcd7d06dd8a"
server_key_iv: "b65027bb0a7b04ece16dd30cbc981503"
server_key_version: "encryptionkey@0"
I20260928 02:55:17.472219 9581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:55:17.474838 9596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:17.476419 9581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:55:17.476557 9581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-5vc0"
server_key: "1d75afc02c3cde4264616dcd7d06dd8a"
server_key_iv: "b65027bb0a7b04ece16dd30cbc981503"
server_key_version: "encryptionkey@0"
I20260928 02:55:17.476675 9581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:17.488479 9581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:17.491811 9581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:17.492004 9581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:17.499089 9581 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:35981
I20260928 02:55:17.499112 9648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:35981 every 8 connection(s)
I20260928 02:55:17.500121 9581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:17.503000 9649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:17.508749 9649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd: Bootstrap starting.
I20260928 02:55:17.510259 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9581
I20260928 02:55:17.510406 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:17.510682 20014 external_mini_cluster.cc:1428] Setting key 375f85ea0616f4684e4b47e7572cf7a0
I20260928 02:55:17.511569 9649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:17.512485 9649 log.cc:824] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:17.514519 9649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd: No bootstrap required, opened a new log
I20260928 02:55:17.518641 9649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35981 } }
I20260928 02:55:17.518900 9649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:17.518955 9649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfda75bf4fd467b8e17d00bb61ed8bd, State: Initialized, Role: FOLLOWER
I20260928 02:55:17.519398 9649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35981 } }
I20260928 02:55:17.519521 9649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:17.519577 9649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:17.519609 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:17.511865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47319 (local address 127.19.139.190:35981)
0928 02:55:17.512314 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:17.512330 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:17.512370 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:55:17.513058 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:17.513080 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:17.513428 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:17.513717 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:17.514046 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:17.515069 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:17.516306 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:17.516795 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:17.516927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:17.517759 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:17.517765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:17.517779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:17.518213 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:17.518269 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:17.518291 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:17.518371 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:17.518523 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:55:17.518737 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":107,"threads_started":1}
I20260928 02:55:17.519668 9649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:17.520599 9649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35981 } }
I20260928 02:55:17.520925 9649 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfda75bf4fd467b8e17d00bb61ed8bd; no voters:
I20260928 02:55:17.521215 9649 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:17.521332 9654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:17.521513 9654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [term 1 LEADER]: Becoming Leader. State: Replica: 7dfda75bf4fd467b8e17d00bb61ed8bd, State: Running, Role: LEADER
I20260928 02:55:17.522145 9654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35981 } }
I20260928 02:55:17.522518 9649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:17.522591 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:17.525550 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35981 } } }
I20260928 02:55:17.525682 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:17.525738 9655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfda75bf4fd467b8e17d00bb61ed8bd. Latest consensus state: current_term: 1 leader_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfda75bf4fd467b8e17d00bb61ed8bd" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35981 } } }
I20260928 02:55:17.525856 9655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd [sys.catalog]: This master's current role is: LEADER
W20260928 02:55:17.527583 9672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:17.527717 9672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:17.527817 9675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:17.528789 9675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:17.529222 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:17.525031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39277 (local address 127.19.139.190:35981)
0928 02:55:17.525123 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:17.525127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:17.525312 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:55:17.525478 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:17.525481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:17.525523 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:17.525588 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:17.525893 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:17.526496 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:17.527620 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:17.527944 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:17.528052 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:17.528837 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:17.528841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:17.528843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:17.528856 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:17.528887 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:17.528900 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:17.528962 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:17.529039 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:55:17.529092 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:55:17.535367 9675 catalog_manager.cc:1347] Generated new cluster ID: acfe399f3f2f41acac7b4a02994c39cf
I20260928 02:55:17.535447 9675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:17.574293 9675 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:17.575253 9675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:17.581645 9675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dfda75bf4fd467b8e17d00bb61ed8bd: Generated new TSK 0
I20260928 02:55:17.582535 9675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:17.594317 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9581
2026-09-28T02:55:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:17 dist-test-slave-5vc0 krb5kdc[9680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:17 dist-test-slave-5vc0 krb5kdc[9680](info): set up 2 sockets
Sep 28 02:55:17 dist-test-slave-5vc0 krb5kdc[9680](info): commencing operation
krb5kdc: starting...
W20260928 02:55:18.544292 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:18 dist-test-slave-5vc0 krb5kdc[9680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:18.666709 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35691
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:18.776597 9693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:18.776877 9693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:18.776919 9693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:18.780970 9693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:18.781044 9693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:18.781062 9693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:18.781092 9693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:18.781121 9693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:18.786639 9693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:18.787735 9693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:18.788787 9693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:18.797003 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:18.796995 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:18.797008 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:18.797101 9693 server_base.cc:1034] running on GCE node
I20260928 02:55:18.797804 9693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:18.798977 9693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:18.800134 9693 hybrid_clock.cc:648] HybridClock initialized: now 1790564118800117 us; error 30 us; skew 500 ppm
Sep 28 02:55:18 dist-test-slave-5vc0 krb5kdc[9680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564118, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:18.802986 9693 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:18.803922 9693 webserver.cc:460] Webserver started at http://127.19.139.190:32811/ using document root <none> and password file <none>
I20260928 02:55:18.804514 9693 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:18.804584 9693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:18.804793 9693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:18.806854 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "c521405fa095472e8a9dd0784a13cf15"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-5vc0"
server_key: "eada24c2db2ba4f401c23cca13e2502c"
server_key_iv: "f8c738624178af1d5eee2b9c998c96bd"
server_key_version: "encryptionkey@0"
I20260928 02:55:18.807415 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "c521405fa095472e8a9dd0784a13cf15"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-5vc0"
server_key: "eada24c2db2ba4f401c23cca13e2502c"
server_key_iv: "f8c738624178af1d5eee2b9c998c96bd"
server_key_version: "encryptionkey@0"
I20260928 02:55:18.811296 9693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:55:18.813915 9708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:18.815227 9693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:55:18.815375 9693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "c521405fa095472e8a9dd0784a13cf15"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-5vc0"
server_key: "eada24c2db2ba4f401c23cca13e2502c"
server_key_iv: "f8c738624178af1d5eee2b9c998c96bd"
server_key_version: "encryptionkey@0"
I20260928 02:55:18.815485 9693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:18.820247 9693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:18.823135 9693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:18.823316 9693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:18.830196 9693 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:35691
I20260928 02:55:18.830258 9760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:35691 every 8 connection(s)
I20260928 02:55:18.831182 9693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:18.832626 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9693
I20260928 02:55:18.832762 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:18.833060 20014 external_mini_cluster.cc:1428] Setting key c0f00ee8f1018ede2be816e039c87a06
I20260928 02:55:18.834561 9761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:18.839114 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:18.834668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58463 (local address 127.19.139.190:35691)
0928 02:55:18.835117 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:18.835136 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:18.835178 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:55:18.836031 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:18.836055 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:18.836495 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:18.836883 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:18.837638 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:18.837644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:18.837661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:18.837673 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:18.837740 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:18.837768 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:18.837920 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:18.837990 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:55:18.838151 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":138,"threads_started":1}
I20260928 02:55:18.841491 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15: Bootstrap starting.
I20260928 02:55:18.842054 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:18.844237 9761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:18.845142 9761 log.cc:824] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:18.846032 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:18.844301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60719 (local address 127.19.139.190:35691)
0928 02:55:18.844424 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:18.844429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:18.844617 (+ 188us) server_negotiation.cc:407] Connection header received
0928 02:55:18.844778 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:18.844781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:18.844820 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:18.844899 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:18.845635 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:18.845638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:18.845641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:55:18.845653 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:18.845679 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:55:18.845699 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:18.845791 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:18.845873 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:55:18.845913 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:55:18.847378 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15: No bootstrap required, opened a new log
I20260928 02:55:18.851020 9761 raft_consensus.cc:348] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c521405fa095472e8a9dd0784a13cf15" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35691 } }
I20260928 02:55:18.851276 9761 raft_consensus.cc:374] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:18.851344 9761 raft_consensus.cc:729] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c521405fa095472e8a9dd0784a13cf15, State: Initialized, Role: FOLLOWER
I20260928 02:55:18.851754 9761 consensus_queue.cc:260] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c521405fa095472e8a9dd0784a13cf15" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35691 } }
I20260928 02:55:18.851881 9761 raft_consensus.cc:388] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:18.851945 9761 raft_consensus.cc:482] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:18.852043 9761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:18.853037 9761 raft_consensus.cc:504] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c521405fa095472e8a9dd0784a13cf15" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35691 } }
I20260928 02:55:18.853377 9761 leader_election.cc:302] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c521405fa095472e8a9dd0784a13cf15; no voters:
I20260928 02:55:18.853693 9761 leader_election.cc:288] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:18.853881 9771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:18.854111 9771 raft_consensus.cc:686] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [term 1 LEADER]: Becoming Leader. State: Replica: c521405fa095472e8a9dd0784a13cf15, State: Running, Role: LEADER
I20260928 02:55:18.854473 9771 consensus_queue.cc:237] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c521405fa095472e8a9dd0784a13cf15" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35691 } }
I20260928 02:55:18.855110 9761 sys_catalog.cc:564] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:18.856233 9773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c521405fa095472e8a9dd0784a13cf15. Latest consensus state: current_term: 1 leader_uuid: "c521405fa095472e8a9dd0784a13cf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c521405fa095472e8a9dd0784a13cf15" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35691 } } }
I20260928 02:55:18.856367 9773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:18.856570 9772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c521405fa095472e8a9dd0784a13cf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c521405fa095472e8a9dd0784a13cf15" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 35691 } } }
I20260928 02:55:18.856753 9772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:18.857025 9782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:18.860034 9782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:18.865854 9782 catalog_manager.cc:1347] Generated new cluster ID: cb7be5840bbe4ec9869dbafe0d7c1768
I20260928 02:55:18.865944 9782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:18.886438 9782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:18.886831 9782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:18.898487 9782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c521405fa095472e8a9dd0784a13cf15: Generated new TSK 0
I20260928 02:55:18.899441 9782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:18.914886 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9693
2026-09-28T02:55:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1320 ms)
[----------] 18 tests from AuthTokenIssuingTest (23815 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:18 dist-test-slave-5vc0 krb5kdc[9792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:18 dist-test-slave-5vc0 krb5kdc[9792](info): set up 2 sockets
Sep 28 02:55:18 dist-test-slave-5vc0 krb5kdc[9792](info): commencing operation
krb5kdc: starting...
W20260928 02:55:19.872146 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:19 dist-test-slave-5vc0 krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:19.993637 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37233
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37233,127.19.139.189:33039,127.19.139.188:32995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:20.103304 9805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:55:20.103619 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:20.103642 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:20.103686 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:20.107915 9805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:20.108000 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:20.108021 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:20.108036 9805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:20.108052 9805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:20.113657 9805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37233,127.19.139.189:33039,127.19.139.188:32995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:20.114774 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:20.115877 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:20.124490 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.124536 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.124567 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:20.124666 9805 server_base.cc:1034] running on GCE node
I20260928 02:55:20.125351 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:20.126492 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:20.127663 9805 hybrid_clock.cc:648] HybridClock initialized: now 1790564120127649 us; error 31 us; skew 500 ppm
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:20.133912 9805 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:20.134889 9805 webserver.cc:460] Webserver started at http://127.19.139.190:39741/ using document root <none> and password file <none>
I20260928 02:55:20.135423 9805 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:20.135481 9805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:20.135650 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:20.137801 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "d50eb3be3d004300a7ae1dbaeec95e8e"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "ffa59799ccdfdb8226957f166a432ab5"
server_key_iv: "b16684cab96ef0ac3ff13db66429d7dc"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.138335 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "d50eb3be3d004300a7ae1dbaeec95e8e"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "ffa59799ccdfdb8226957f166a432ab5"
server_key_iv: "b16684cab96ef0ac3ff13db66429d7dc"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.142057 9805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 02:55:20.144449 9820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:20.145677 9805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:55:20.145781 9805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "d50eb3be3d004300a7ae1dbaeec95e8e"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "ffa59799ccdfdb8226957f166a432ab5"
server_key_iv: "b16684cab96ef0ac3ff13db66429d7dc"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.145900 9805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:20.158794 9805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:20.161623 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:20.161773 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:20.168269 9805 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:37233
I20260928 02:55:20.168547 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:37233 every 8 connection(s)
I20260928 02:55:20.169298 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:20.169819 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9805
I20260928 02:55:20.169956 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:20.170269 20014 external_mini_cluster.cc:1428] Setting key d58fbdb3e6f5f1a80cbf553c4069009f
I20260928 02:55:20.172712 9873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:20.177470 9873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.139.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.189@KRBTEST.COM" created.
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:55:20.193418 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.179840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.19.139.190:37233)
0928 02:55:20.180361 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.180379 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.180562 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:55:20.181710 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.181768 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.181980 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.182143 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.183079 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.184418 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.185449 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.186057 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.186227 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.188542 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.188549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.188563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.188589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.190106 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.190548 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.190552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.190554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.190659 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.191216 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.191219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.191221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.191616 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.191743 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.192073 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.192336 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":391,"spinlock_wait_cycles":50816,"thread_start_us":141,"threads_started":1}
I20260928 02:55:20.193418 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.179848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37233 (local address 127.0.0.1:53320)
0928 02:55:20.180374 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.180424 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.181370 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.182185 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.182192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.182671 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.182936 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.182950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.184523 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.184526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.185316 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.185325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.186195 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.186198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.186272 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.186917 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.186933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.188401 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.190210 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.190219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.190228 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.190466 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.191013 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.191015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.191017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.191127 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.191717 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.191724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.191852 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.191999 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.192396 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":52,"spinlock_wait_cycles":6144,"thread_start_us":130,"threads_started":1}
I20260928 02:55:20.195941 9873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab.
Entry for principal kudu/127.19.139.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab.
I20260928 02:55:20.196810 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.196478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37910)
0928 02:55:20.196653 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.196728 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260928 02:55:20.197009 9823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
I20260928 02:55:20.197331 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:33039
--webserver_interface=127.19.139.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab
--principal=kudu/127.19.139.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37233,127.19.139.189:33039,127.19.139.188:32995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:20.197841 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.197637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37912)
0928 02:55:20.197736 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.197783 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:55:20.197963 9873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:33039: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
I20260928 02:55:20.247262 9873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } attempt: 1
I20260928 02:55:20.248241 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.247847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37918)
0928 02:55:20.248032 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.248100 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:55:20.248996 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.248785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37922)
0928 02:55:20.248885 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.248933 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:55:20.249112 9873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:33039: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
W20260928 02:55:20.307634 9880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:55:20.307979 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:20.308019 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:20.308084 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:20.312026 9880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:20.312085 9880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:20.312110 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:20.312131 9880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:20.312179 9880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:20.317281 9873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } attempt: 2
I20260928 02:55:20.317489 9880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37233,127.19.139.189:33039,127.19.139.188:32995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab
--principal=kudu/127.19.139.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.139.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:20.318106 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.317806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37926)
0928 02:55:20.317970 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.318032 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:55:20.318763 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:20.318827 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.318597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37928)
0928 02:55:20.318704 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.318762 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:55:20.318957 9873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:33039: Network error: Client connection negotiation failed: client connection to 127.19.139.189:33039: connect: Connection refused (error 111)
I20260928 02:55:20.319859 9880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:20.327966 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.327996 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:20.328140 9880 server_base.cc:1034] running on GCE node
W20260928 02:55:20.327975 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:20.328832 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:20.329952 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:20.331111 9880 hybrid_clock.cc:648] HybridClock initialized: now 1790564120331084 us; error 43 us; skew 500 ppm
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:20.333796 9880 init.cc:377] Logged in from keytab as kudu/127.19.139.189@KRBTEST.COM (short username kudu)
I20260928 02:55:20.334712 9880 webserver.cc:460] Webserver started at http://127.19.139.189:34897/ using document root <none> and password file <none>
I20260928 02:55:20.335283 9880 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:20.335347 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:20.335561 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:20.337718 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data/instance:
uuid: "380f3a4167ac4a34ab9de9985ef2b6c3"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "59feef27122322e5e66fd624dcdc56ae"
server_key_iv: "2f67fd05df76aea42e64f797dd62a2e4"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.338295 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal/instance:
uuid: "380f3a4167ac4a34ab9de9985ef2b6c3"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "59feef27122322e5e66fd624dcdc56ae"
server_key_iv: "2f67fd05df76aea42e64f797dd62a2e4"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.342139 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:55:20.344821 9895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:20.346072 9880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:20.346195 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal
uuid: "380f3a4167ac4a34ab9de9985ef2b6c3"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "59feef27122322e5e66fd624dcdc56ae"
server_key_iv: "2f67fd05df76aea42e64f797dd62a2e4"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.346304 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:20.362473 9880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:20.365645 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:20.365839 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:20.373031 9880 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.189:33039
I20260928 02:55:20.373306 9947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.189:33039 every 8 connection(s)
I20260928 02:55:20.374099 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
I20260928 02:55:20.377051 9948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:20.380849 9948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:20.383895 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9880
I20260928 02:55:20.384037 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-1/wal/instance
I20260928 02:55:20.384321 20014 external_mini_cluster.cc:1428] Setting key 73d4c50d380908cfcc45fc0ef6f67c84
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.188@KRBTEST.COM" created.
I20260928 02:55:20.394932 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.382746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.19.139.190:37233)
0928 02:55:20.382876 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.382879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.383791 (+ 912us) server_negotiation.cc:407] Connection header received
0928 02:55:20.384883 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.384887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.384950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.385022 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.386220 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.386760 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.387953 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.388219 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.388308 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.391081 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.391085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.391086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.391117 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.392712 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.393264 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.393268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.393270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.393355 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.393672 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.393676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.393677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.393941 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.394033 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.394618 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.394762 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:55:20.395816 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.383098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37233 (local address 127.0.0.1:53324)
0928 02:55:20.383622 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.383668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.384673 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.385219 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.385225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.385578 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.386071 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.386085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.386887 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.386891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.387831 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.387841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.388372 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.388376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.388544 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.389177 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.389195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.390826 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.392858 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.392864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.392878 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.393165 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.393439 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.393442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.393443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.393587 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.394082 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.394088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.394370 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.394547 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.394910 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":153,"threads_started":1}
I20260928 02:55:20.397341 9948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:20.408025 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.397885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37942)
0928 02:55:20.397995 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.398009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.398091 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.398784 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.398787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.398932 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.399025 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.399030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.400475 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.400478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.401116 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.401125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.401911 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.401914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.401994 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.402442 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.402454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.404496 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.406549 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.406553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.406555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.406784 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.407067 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.407071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.407073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.407174 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.407604 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.407607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.407698 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.407788 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.407897 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:55:20.408219 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.397970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.19.139.189:33039)
0928 02:55:20.398322 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.398334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.398359 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:55:20.398404 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.398410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.398637 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.398802 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.399141 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.400359 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.401235 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.401803 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.401891 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.404634 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.404643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.404655 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.404676 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.406453 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.406886 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.406891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.406892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.406967 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.407271 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.407273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.407274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.407526 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.407627 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.407889 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.408011 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":141,"threads_started":1}
I20260928 02:55:20.409950 9948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:20.410121 9873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab.
Entry for principal kudu/127.19.139.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab.
I20260928 02:55:20.410728 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.410417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47052)
0928 02:55:20.410577 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.410656 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 02:55:20.411028 9896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
I20260928 02:55:20.411197 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:32995
--webserver_interface=127.19.139.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab
--principal=kudu/127.19.139.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:37233,127.19.139.189:33039,127.19.139.188:32995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:20.412042 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.411826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47062)
0928 02:55:20.411932 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.411982 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:55:20.412240 9948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:32995: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:20.418809 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.410537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.19.139.189:33039)
0928 02:55:20.410897 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.410900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.410911 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:55:20.410940 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.410942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.410977 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.411035 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.411467 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.412126 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.413077 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.413912 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.414005 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.416042 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.416046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.416048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.416071 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.417370 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.417750 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.417753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.417755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.417834 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.418072 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.418075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.418077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.418239 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.418319 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.418558 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.418657 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260928 02:55:20.418804 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.410530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37952)
0928 02:55:20.410707 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.410719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.410815 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.411074 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.411077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.411233 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.411338 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.411346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.412310 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.412315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.412949 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.412960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.414025 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.414029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.414139 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.414572 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.414586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.415921 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.417472 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.417476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.417478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.417667 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.417913 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.417916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.417918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.417996 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.418324 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.418328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.418419 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.418501 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.418639 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 02:55:20.419520 9873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:20.420578 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.420360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47078)
0928 02:55:20.420467 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.420518 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:55:20.421182 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.420968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47088)
0928 02:55:20.421091 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.421134 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 02:55:20.421290 9873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:32995: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
I20260928 02:55:20.452445 9873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } attempt: 1
I20260928 02:55:20.453270 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.452978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47102)
0928 02:55:20.453119 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.453189 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:55:20.453969 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.453764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47112)
0928 02:55:20.453868 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.453911 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:55:20.454082 9873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:32995: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
I20260928 02:55:20.461517 9948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } attempt: 1
I20260928 02:55:20.462365 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.462090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47122)
0928 02:55:20.462233 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.462292 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:55:20.463032 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.462832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47138)
0928 02:55:20.462931 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.462976 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:55:20.463146 9948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:32995: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
W20260928 02:55:20.521807 9955 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:55:20.522112 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:20.522147 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:20.522212 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:20.526324 9955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:20.526393 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:20.526419 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:20.526443 9955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:20.526468 9955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:20.529247 9873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } attempt: 2
I20260928 02:55:20.530040 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.529747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47154)
0928 02:55:20.529890 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.529952 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:55:20.530762 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.530500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47164)
0928 02:55:20.530604 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.530648 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:55:20.530879 9873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:32995: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
I20260928 02:55:20.531323 9948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } attempt: 2
I20260928 02:55:20.531924 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.531687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47178)
0928 02:55:20.531811 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.531859 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:55:20.532140 9955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:37233,127.19.139.189:33039,127.19.139.188:32995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab
--principal=kudu/127.19.139.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:32995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.139.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:20.532640 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.532403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:47180)
0928 02:55:20.532517 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.532574 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:55:20.532860 9948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:32995: Network error: Client connection negotiation failed: client connection to 127.19.139.188:32995: connect: Connection refused (error 111)
I20260928 02:55:20.533422 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:20.534425 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:55:20.542991 9955 server_base.cc:1034] running on GCE node
W20260928 02:55:20.542930 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.542928 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.542945 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:20.543756 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:20.544893 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:20.546104 9955 hybrid_clock.cc:648] HybridClock initialized: now 1790564120546083 us; error 36 us; skew 500 ppm
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:20.549118 9955 init.cc:377] Logged in from keytab as kudu/127.19.139.188@KRBTEST.COM (short username kudu)
I20260928 02:55:20.550154 9955 webserver.cc:460] Webserver started at http://127.19.139.188:39271/ using document root <none> and password file <none>
I20260928 02:55:20.550769 9955 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:20.550848 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:20.551055 9955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:20.553180 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data/instance:
uuid: "7743ba0dbd4a40d4bddbd363216eb888"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "98148827cdee4594816cfd108598d191"
server_key_iv: "37dc41a89ab057c76b67d849503c8eea"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.553736 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal/instance:
uuid: "7743ba0dbd4a40d4bddbd363216eb888"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "98148827cdee4594816cfd108598d191"
server_key_iv: "37dc41a89ab057c76b67d849503c8eea"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.557555 9955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:55:20.560357 9970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:20.561702 9955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:55:20.561873 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal
uuid: "7743ba0dbd4a40d4bddbd363216eb888"
format_stamp: "Formatted at 2026-09-28 02:55:20 on dist-test-slave-5vc0"
server_key: "98148827cdee4594816cfd108598d191"
server_key_iv: "37dc41a89ab057c76b67d849503c8eea"
server_key_version: "encryptionkey@0"
I20260928 02:55:20.561986 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:20.593503 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:20.596798 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:20.596988 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:20.603751 9955 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.188:32995
I20260928 02:55:20.604003 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.188:32995 every 8 connection(s)
I20260928 02:55:20.604784 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
I20260928 02:55:20.607656 10023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:20.608103 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 9955
I20260928 02:55:20.608239 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563845681472-20014-0/minicluster-data/master-2/wal/instance
I20260928 02:55:20.608498 20014 external_mini_cluster.cc:1428] Setting key b23ea20de7c46fbeab46d73aafb2fbbb
I20260928 02:55:20.612608 10023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:55:20.623384 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.609915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.19.139.188:32995)
0928 02:55:20.610374 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.610389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.610428 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:55:20.611119 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.611137 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.611487 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.611892 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.612748 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.613924 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.615099 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.615653 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.615773 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.618657 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.618663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.618673 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.618698 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.620441 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.620846 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.620851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.620856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.620957 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.621203 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.621206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.621208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.621589 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.621761 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.622054 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.622340 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260928 02:55:20.623903 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.614387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.19.139.190:37233)
0928 02:55:20.614535 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.614540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.614882 (+ 342us) server_negotiation.cc:407] Connection header received
0928 02:55:20.615030 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.615033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.615090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.615180 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.615963 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.616493 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.617784 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.618118 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.618226 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.620606 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.620610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.620612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.620638 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.621905 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.622387 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.622391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.622393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.622482 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.622797 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.622801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.622803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.623037 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.623134 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.623601 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.623741 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:55:20.623988 9948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } attempt: 3
I20260928 02:55:20.624615 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.614327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:37233 (local address 127.0.0.1:58660)
0928 02:55:20.614746 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.614800 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.614924 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.615259 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.615264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.615611 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.615836 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.615844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.616613 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.616616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.617652 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.617668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.618281 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.618285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.618380 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.618851 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.618863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.620465 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.622007 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.622015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.622020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.622267 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.622589 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.622592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.622594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.622676 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.623192 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.623199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.623336 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.624105 (+ 769us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.624357 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":139,"threads_started":1}
I20260928 02:55:20.625837 10023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:20.630036 9873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } attempt: 3
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:20.635625 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.624561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:33512)
0928 02:55:20.624721 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.624746 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.624856 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.626635 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.626639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.626833 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.626965 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.626974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.627943 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.627947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.628729 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.628746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.629297 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.629301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.629909 (+ 608us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.630400 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.630415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.631888 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.634088 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.634092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.634095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.634298 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.634597 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.634600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.634602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.634692 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.635142 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.635145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.635233 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.635311 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.635439 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":527}
I20260928 02:55:20.636324 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.624540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.19.139.188:32995)
0928 02:55:20.626387 (+ 1847us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.626391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.626403 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:55:20.626434 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.626437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.626478 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.626543 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.627092 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.627810 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.628860 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.629173 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.629384 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.632028 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.632032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.632034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.632057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.633980 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.634386 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.634389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.634391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.634492 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.634813 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.634816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.634818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.635031 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.635929 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.636004 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.636117 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786,"thread_start_us":109,"threads_started":1}
I20260928 02:55:20.637533 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.626282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.19.139.190:37233)
0928 02:55:20.627068 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.627072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.627085 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:55:20.627118 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.627121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.627162 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.627234 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.628047 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.628814 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.629811 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.630047 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.630138 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.633017 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.633021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.633024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.633051 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.635246 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.635742 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.635746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.635748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.635841 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.636142 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.636145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.636147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.636384 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.636630 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.637265 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.637384 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712}
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:55:20.638360 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.626905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:33039 (local address 127.0.0.1:37676)
0928 02:55:20.628299 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.628312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.628466 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.628892 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.628895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.629082 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.629178 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.629183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.630125 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.630128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.632136 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.632148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.632874 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.632878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.632979 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.633326 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.633363 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.634750 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.636673 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.636675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.636677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.636907 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.637168 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.637171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.637172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.637270 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.637903 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.637906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.637994 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.638094 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.638211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291}
I20260928 02:55:20.638445 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.627065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.19.139.189:33039)
0928 02:55:20.627203 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.627236 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.628383 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 02:55:20.628584 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.628587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.628654 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.628724 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.629293 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.629815 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.632320 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.632636 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.632714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.634879 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.634882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.634885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.634906 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.636580 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.636996 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.636999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.637001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.637073 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.637365 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.637368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.637370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.637594 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.637660 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.638185 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.638300 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260928 02:55:20.640213 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: Bootstrap starting.
I20260928 02:55:20.640347 10023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:20.641677 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.630415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:33522)
0928 02:55:20.630533 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.630829 (+ 296us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.630904 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.631165 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.631168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.631339 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.631433 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.631439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.633084 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.633087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.634483 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.634497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.635405 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.635409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.635494 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.636086 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.636101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.638166 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.640054 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.640057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.640058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.640308 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.640602 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.640605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.640607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.640776 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.641179 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.641182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.641307 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.641406 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.641531 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:55:20.642273 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.630637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.19.139.188:32995)
0928 02:55:20.630841 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.630856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.630868 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:55:20.631001 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.631004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.631046 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.631538 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.631715 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.632772 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.634698 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.635048 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.638304 (+ 3256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.638406 (+ 102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.638409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.638411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.638426 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.639947 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.640417 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.640420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.640422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.640505 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.640874 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.640876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.640878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.641094 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.641188 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.641981 (+ 793us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.642098 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 02:55:20.643639 9948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:20.644572 9948 log.cc:824] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:55:20.646899 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: Bootstrap starting.
I20260928 02:55:20.646960 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: No bootstrap required, opened a new log
I20260928 02:55:20.649852 9873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:20.650105 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.640876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:32995 (local address 127.0.0.1:33536)
0928 02:55:20.640971 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:20.640984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:20.641053 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:20.642782 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:20.642784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:20.642926 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:20.643001 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.643007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.644000 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.644003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.644760 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:20.644771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.645248 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.645251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:20.645325 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.645766 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:20.645779 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:20.646879 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:20.648496 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.648498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.648500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.648720 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.648957 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:20.648960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:20.648961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:20.649034 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:20.649500 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:20.649502 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:20.649584 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 02:55:20.649663 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:55:20.649794 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:55:20.650789 9873 log.cc:824] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:20.651015 9948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.651268 9948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380f3a4167ac4a34ab9de9985ef2b6c3, State: Initialized, Role: FOLLOWER
I20260928 02:55:20.651427 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.641059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.19.139.188:32995)
0928 02:55:20.642560 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.642564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.642574 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:55:20.642600 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.642603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.642638 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.642702 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.643121 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.643897 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.644860 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.645134 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.645524 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.647035 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.647039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.647041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.647064 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.648387 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.648808 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.648810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.648812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.648879 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.649117 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.649119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.649121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.649306 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.650731 (+ 1425us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.650807 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.651133 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455}
I20260928 02:55:20.651633 9948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
Sep 28 02:55:20 dist-test-slave-5vc0 krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:20.653057 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: No bootstrap required, opened a new log
I20260928 02:55:20.653085 9948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:20.654902 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: Bootstrap starting.
I20260928 02:55:20.657234 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.641296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.19.139.189:33039)
0928 02:55:20.641393 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.641397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.641408 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:55:20.641497 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.641500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.641547 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.641608 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.642266 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.648829 (+ 6563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.649938 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.650264 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.650351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.653260 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.653264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.653267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.653294 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.654873 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.655490 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.655493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.655495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.655584 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.655867 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.655870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.655872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.656088 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.656885 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.656960 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.657076 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:55:20.657446 9873 raft_consensus.cc:348] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.657665 9873 raft_consensus.cc:729] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50eb3be3d004300a7ae1dbaeec95e8e, State: Initialized, Role: FOLLOWER
I20260928 02:55:20.657781 10023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:20.657985 9873 consensus_queue.cc:260] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.658672 10023 log.cc:824] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:20.659631 9873 sys_catalog.cc:564] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:20.656781 9880 master_runner.cc:186] Error getting master registration for 127.19.139.190:37233: OK, instance_id { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" instance_seqno: 1790564120166476 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:55:20.660573 9880 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:20.660825 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:20.660967 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.661088 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:20.662077 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.662230 9847 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662
I20260928 02:55:20.662681 9847 raft_consensus.cc:482] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:55:20.662817 9847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:55:20.663300 9826 tablet.cc:2369] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:55:20.664090 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: No bootstrap required, opened a new log
I20260928 02:55:20.664098 9847 raft_consensus.cc:504] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.665153 9847 leader_election.cc:288] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [CANDIDATE]: Term 1 election: Requested vote from peers 380f3a4167ac4a34ab9de9985ef2b6c3 (127.19.139.189:33039), 7743ba0dbd4a40d4bddbd363216eb888 (127.19.139.188:32995)
W20260928 02:55:20.665547 10055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:20.665679 10055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:20.667898 9922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3"
I20260928 02:55:20.668148 9922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:20.668280 10023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.668531 10023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7743ba0dbd4a40d4bddbd363216eb888, State: Initialized, Role: FOLLOWER
I20260928 02:55:20.668877 10023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.669582 9922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50eb3be3d004300a7ae1dbaeec95e8e in term 1.
I20260928 02:55:20.669997 9823 leader_election.cc:302] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380f3a4167ac4a34ab9de9985ef2b6c3, d50eb3be3d004300a7ae1dbaeec95e8e; no voters:
I20260928 02:55:20.670336 10023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:20.670372 9997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7743ba0dbd4a40d4bddbd363216eb888"
I20260928 02:55:20.670624 9997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:20.671969 9997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50eb3be3d004300a7ae1dbaeec95e8e in term 1.
I20260928 02:55:20.672796 10058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:20.673070 10058 raft_consensus.cc:686] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [term 1 LEADER]: Becoming Leader. State: Replica: d50eb3be3d004300a7ae1dbaeec95e8e, State: Running, Role: LEADER
I20260928 02:55:20.673496 10058 consensus_queue.cc:237] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } }
I20260928 02:55:20.675588 10058 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50eb3be3d004300a7ae1dbaeec95e8e. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.671900 9805 master_runner.cc:186] Error getting master registration for 127.19.139.188:32995: OK, instance_id { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" instance_seqno: 1790564120601913 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 02:55:20.675820 10042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:20.675901 10042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:20.676033 9805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:20.676141 10058 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:20.677492 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:20.679148 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.679275 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:55:20.679919 10075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:20.679988 10075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:20.690104 10069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:20.691084 10069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:20.691485 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.678126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.19.139.188:32995)
0928 02:55:20.678240 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.678244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.680067 (+ 1823us) server_negotiation.cc:407] Connection header received
0928 02:55:20.680277 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.680281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.680329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.680377 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.681354 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.682091 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.683246 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.683577 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.683654 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.685612 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.685616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.685618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.685644 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.687295 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.688374 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.688378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.688380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.688472 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.688776 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.688779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.688780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.688989 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.689110 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.691204 (+ 2094us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.691327 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:55:20.695013 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.679637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.19.139.189:33039)
0928 02:55:20.679772 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.679777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.684826 (+ 5049us) server_negotiation.cc:407] Connection header received
0928 02:55:20.685472 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.685475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.685524 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.685571 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.686334 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.687105 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.688265 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.688602 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.688676 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.690300 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.690305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.690307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.690336 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.691922 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.693652 (+ 1730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.693656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.693658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.693759 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.694062 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.694065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.694067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.694283 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.694393 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.694719 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.694852 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:55:20.698004 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:20.678804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.19.139.190:37233)
0928 02:55:20.678904 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.678908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.684611 (+ 5703us) server_negotiation.cc:407] Connection header received
0928 02:55:20.685076 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.685079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.685130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.685178 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.689429 (+ 4251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.690072 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.692206 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:20.692496 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:20.692572 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:20.694224 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:20.694228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:20.694231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:20.694259 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:20.696110 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.696704 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.696708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.696710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.696795 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:20.697069 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:20.697072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:20.697074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:20.697291 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:20.697387 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:20.697726 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:55:20.697843 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:55:20.701459 9921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [term 1 FOLLOWER]: Refusing update from remote peer d50eb3be3d004300a7ae1dbaeec95e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:20.701579 9997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [term 1 FOLLOWER]: Refusing update from remote peer d50eb3be3d004300a7ae1dbaeec95e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:20.702620 10064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:20.703105 10064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:20.707336 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50eb3be3d004300a7ae1dbaeec95e8e. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.707504 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:20.708062 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50eb3be3d004300a7ae1dbaeec95e8e. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.708215 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:20.712437 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.712584 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:20.713847 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.713981 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:20.714576 10082 mvcc.cc:205] Tried to move back new op lower bound from 7334150638381228032 to 7334150638282997760. Current Snapshot: MvccSnapshot[applied={T|T < 7334150638381228032}]
I20260928 02:55:20.715665 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.715793 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:20.715991 10064 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d50eb3be3d004300a7ae1dbaeec95e8e" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 37233 } } peers { permanent_uuid: "380f3a4167ac4a34ab9de9985ef2b6c3" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 33039 } } peers { permanent_uuid: "7743ba0dbd4a40d4bddbd363216eb888" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 32995 } } }
I20260928 02:55:20.716102 10064 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:20.716379 10084 mvcc.cc:205] Tried to move back new op lower bound from 7334150638381228032 to 7334150638282997760. Current Snapshot: MvccSnapshot[applied={T|T < 7334150638381228032}]
I20260928 02:55:20.717495 10069 catalog_manager.cc:1347] Generated new cluster ID: 82aa1e02ed704e81bf6672468d253c5d
I20260928 02:55:20.717552 10069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:20.719403 10083 mvcc.cc:205] Tried to move back new op lower bound from 7334150638381228032 to 7334150638282997760. Current Snapshot: MvccSnapshot[applied={T|T < 7334150638381228032}]
I20260928 02:55:20.745447 10069 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:20.746797 10069 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:20.769086 10069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d50eb3be3d004300a7ae1dbaeec95e8e: Generated new TSK 0
I20260928 02:55:20.770043 10069 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:55:20.808139 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.805260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.19.139.189:33039)
0928 02:55:20.805392 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.805397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.806940 (+ 1543us) server_negotiation.cc:407] Connection header received
0928 02:55:20.807142 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.807145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.807201 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.807258 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.807987 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37704: BlockingRecv error: recv got EOF from 127.0.0.1:37704 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:55:20.809686 9877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.806091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.19.139.190:37233)
0928 02:55:20.806213 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.806217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.808704 (+ 2487us) server_negotiation.cc:407] Connection header received
0928 02:55:20.808878 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.808881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.808930 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.809030 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.809549 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58682: BlockingRecv error: recv got EOF from 127.0.0.1:58682 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 02:55:20.811610 10028 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.810410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.19.139.188:32995)
0928 02:55:20.810516 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.810520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.810705 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:55:20.810864 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.810867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.810914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.810960 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.811485 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33554: BlockingRecv error: recv got EOF from 127.0.0.1:33554 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 02:55:20.816296 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.814663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.19.139.189:33039)
0928 02:55:20.814778 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.814782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.815245 (+ 463us) server_negotiation.cc:407] Connection header received
0928 02:55:20.815463 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.815467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.815546 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.815602 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.816128 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37706: BlockingRecv error: recv got EOF from 127.0.0.1:37706 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:55:20.821679 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.820007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.19.139.189:33039)
0928 02:55:20.820112 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.820117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.820698 (+ 581us) server_negotiation.cc:407] Connection header received
0928 02:55:20.820907 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.820911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.820966 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.821022 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.821559 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37722: BlockingRecv error: recv got EOF from 127.0.0.1:37722 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:55:20.827383 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.825781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.19.139.189:33039)
0928 02:55:20.825887 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.825891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.826431 (+ 540us) server_negotiation.cc:407] Connection header received
0928 02:55:20.826642 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.826646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.826708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.826763 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.827274 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37734: BlockingRecv error: recv got EOF from 127.0.0.1:37734 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:55:20.835294 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.833687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.19.139.189:33039)
0928 02:55:20.833826 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.833830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.834378 (+ 548us) server_negotiation.cc:407] Connection header received
0928 02:55:20.834563 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.834567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.834618 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.834679 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.835183 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37740: BlockingRecv error: recv got EOF from 127.0.0.1:37740 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:55:20.847645 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.845833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.19.139.189:33039)
0928 02:55:20.845975 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.845980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.846628 (+ 648us) server_negotiation.cc:407] Connection header received
0928 02:55:20.846859 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.846862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.846914 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.846964 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.847520 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37746: BlockingRecv error: recv got EOF from 127.0.0.1:37746 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:55:20.867559 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.866179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.19.139.189:33039)
0928 02:55:20.866307 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.866312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.866432 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:55:20.866678 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.866683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.866746 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.866800 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.867434 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37750: BlockingRecv error: recv got EOF from 127.0.0.1:37750 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 02:55:20.903287 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.902250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.19.139.189:33039)
0928 02:55:20.902414 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.902419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.902526 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:55:20.902646 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.902648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.902698 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.902775 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.903163 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37766: BlockingRecv error: recv got EOF from 127.0.0.1:37766 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:55:20.970570 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:20.969388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.19.139.189:33039)
0928 02:55:20.969551 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:20.969556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:20.969656 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:55:20.969872 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:20.969876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:20.969935 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:20.970004 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:20.970452 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37774: BlockingRecv error: recv got EOF from 127.0.0.1:37774 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:55:21.101862 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:21.100637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.19.139.189:33039)
0928 02:55:21.100818 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.100822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.100971 (+ 149us) server_negotiation.cc:407] Connection header received
0928 02:55:21.101083 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.101086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.101137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.101229 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:21.101761 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37776: BlockingRecv error: recv got EOF from 127.0.0.1:37776 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 02:55:21.361151 9953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:21.359970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.19.139.189:33039)
0928 02:55:21.360117 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.360122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.360227 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:55:21.360406 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.360409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.360462 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.360568 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:21.361008 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37788: BlockingRecv error: recv got EOF from 127.0.0.1:37788 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:55:21.677351 10042 catalog_manager.cc:1259] Loaded cluster ID: 82aa1e02ed704e81bf6672468d253c5d
I20260928 02:55:21.677443 10042 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: loading cluster ID for follower catalog manager: success
I20260928 02:55:21.679147 10042 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: acquiring CA information for follower catalog manager: success
I20260928 02:55:21.680444 10042 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 380f3a4167ac4a34ab9de9985ef2b6c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:55:21.681141 10075 catalog_manager.cc:1259] Loaded cluster ID: 82aa1e02ed704e81bf6672468d253c5d
I20260928 02:55:21.681200 10075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: loading cluster ID for follower catalog manager: success
I20260928 02:55:21.682700 10075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: acquiring CA information for follower catalog manager: success
I20260928 02:55:21.683741 10075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7743ba0dbd4a40d4bddbd363216eb888: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:55:21.879210 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.875223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.19.139.189:33039)
0928 02:55:21.875494 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.875499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.875516 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:55:21.875691 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.875694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.875824 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.875922 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:21.876212 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.876786 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.877683 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.877913 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.878003 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.878345 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:21.878796 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:21.878912 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.878982 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.879046 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
I20260928 02:55:21.886108 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.882358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.19.139.188:32995)
0928 02:55:21.882587 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.882592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.882606 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:55:21.882787 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.882791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.882917 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.883024 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:21.883241 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.883761 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.884642 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.884872 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.884964 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.885258 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:21.885702 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:21.885814 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.885896 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.885957 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260928 02:55:21.893956 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.889405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.19.139.189:33039)
0928 02:55:21.889541 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.889546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.890127 (+ 581us) server_negotiation.cc:407] Connection header received
0928 02:55:21.890210 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.890213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.890265 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.890324 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:21.890569 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.891404 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.892892 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.893122 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.893207 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.893444 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:21.893517 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:21.893572 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.893725 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.893794 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:55:21.894094 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.889603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.19.139.188:32995)
0928 02:55:21.889723 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.889728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.889831 (+ 103us) server_negotiation.cc:407] Connection header received
0928 02:55:21.890049 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.890053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.890111 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.890163 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:21.890610 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.891403 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.892857 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.893188 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.893290 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.893617 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:21.893698 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:21.893756 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.893910 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.893964 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:55:21.894529 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.889405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.19.139.190:37233)
0928 02:55:21.889809 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.889815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.889895 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:55:21.890097 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.890101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.890296 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.890419 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:21.890778 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.891484 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.892602 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.892853 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.892958 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.893400 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:21.893773 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:21.893859 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.894341 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.894403 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":81,"threads_started":1}
I20260928 02:55:21.897532 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9805
I20260928 02:55:21.903505 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9880
I20260928 02:55:21.908742 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 9955
2026-09-28T02:55:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2994 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:21 dist-test-slave-5vc0 krb5kdc[10170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:21 dist-test-slave-5vc0 krb5kdc[10170](info): set up 2 sockets
Sep 28 02:55:21 dist-test-slave-5vc0 krb5kdc[10170](info): commencing operation
krb5kdc: starting...
W20260928 02:55:22.856385 20014 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/test-user.keytab.
Sep 28 02:55:22 dist-test-slave-5vc0 krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.139.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
Entry for principal kudu/127.19.139.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab.
I20260928 02:55:22.977659 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45743
--webserver_interface=127.19.139.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45743,127.19.139.189:37399,127.19.139.188:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:23.087486 10183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:55:23.087793 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:23.087828 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:23.087893 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:23.092037 10183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:23.092115 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:23.092142 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:23.092183 10183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:23.092203 10183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:23.097607 10183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45743,127.19.139.189:37399,127.19.139.188:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.190.keytab
--principal=kudu/127.19.139.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.190:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.139.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:23.098693 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:23.099727 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:23.108417 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:23.108464 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:23.108506 10183 server_base.cc:1034] running on GCE node
W20260928 02:55:23.108474 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:23.109316 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:23.110548 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:23.111716 10183 hybrid_clock.cc:648] HybridClock initialized: now 1790564123111724 us; error 63 us; skew 500 ppm
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:23.114567 10183 init.cc:377] Logged in from keytab as kudu/127.19.139.190@KRBTEST.COM (short username kudu)
I20260928 02:55:23.115469 10183 webserver.cc:460] Webserver started at http://127.19.139.190:42437/ using document root <none> and password file <none>
I20260928 02:55:23.116012 10183 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:23.116079 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:23.116312 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:23.118388 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data/instance:
uuid: "9f6a1e20177940d0a9404f54c3dd0068"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "7be3b5c6458747fcb26fc966a7b4d28d"
server_key_iv: "4177c304f77644d8ce80a689a0ef9746"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.118937 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal/instance:
uuid: "9f6a1e20177940d0a9404f54c3dd0068"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "7be3b5c6458747fcb26fc966a7b4d28d"
server_key_iv: "4177c304f77644d8ce80a689a0ef9746"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.122882 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:55:23.125435 10198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:23.126706 10183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:55:23.126833 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal
uuid: "9f6a1e20177940d0a9404f54c3dd0068"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "7be3b5c6458747fcb26fc966a7b4d28d"
server_key_iv: "4177c304f77644d8ce80a689a0ef9746"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.126938 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:23.152504 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:23.155678 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:23.155861 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:23.162528 10183 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.190:45743
I20260928 02:55:23.162806 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.190:45743 every 8 connection(s)
I20260928 02:55:23.163506 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:23.163826 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 10183
I20260928 02:55:23.163940 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-0/wal/instance
I20260928 02:55:23.164279 20014 external_mini_cluster.cc:1428] Setting key 51c99fec6fad6dd69845e34c8d9ef8a7
I20260928 02:55:23.166313 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:23.170167 10251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.139.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.189@KRBTEST.COM" created.
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
I20260928 02:55:23.187278 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.172637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.19.139.190:45743)
0928 02:55:23.173216 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.173245 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.173866 (+ 621us) server_negotiation.cc:407] Connection header received
0928 02:55:23.175185 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.175191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.175457 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.175646 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.176459 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.177365 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.178802 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.179182 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.179296 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.181913 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.181919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.181930 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.181953 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.183588 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.184250 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.184255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.184257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.184420 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.184708 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.184710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.184711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.185203 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.185352 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.186624 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.186780 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":30464,"thread_start_us":170,"threads_started":1}
I20260928 02:55:23.187284 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.172637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45743 (local address 127.0.0.1:50236)
0928 02:55:23.173637 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.173696 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.174811 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.175662 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.175666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.176004 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.176304 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.176320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.177496 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.177499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.178686 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.178699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.179345 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.179350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.179458 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.180061 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.180084 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.181760 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.183740 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.183748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.183757 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.184029 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.184501 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.184504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.184505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.184630 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.185380 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.185388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.185560 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.185841 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.186201 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":161,"threads_started":1}
I20260928 02:55:23.190367 10251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab.
Entry for principal kudu/127.19.139.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab.
I20260928 02:55:23.191323 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.190969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48482)
0928 02:55:23.191122 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.191237 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:55:23.191532 10202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
I20260928 02:55:23.192023 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:37399
--webserver_interface=127.19.139.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab
--principal=kudu/127.19.139.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45743,127.19.139.189:37399,127.19.139.188:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:23.192541 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.192304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48494)
0928 02:55:23.192416 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.192474 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:55:23.192699 10251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:37399: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
I20260928 02:55:23.241997 10251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } attempt: 1
I20260928 02:55:23.242969 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.242602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48500)
0928 02:55:23.242817 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.242884 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 02:55:23.243731 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.243479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48512)
0928 02:55:23.243615 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.243669 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:55:23.243896 10251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:37399: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
W20260928 02:55:23.302752 10258 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:55:23.303058 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:23.303082 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:23.303129 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:23.307286 10258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:23.307363 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:23.307382 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:23.307397 10258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:23.307412 10258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:23.312106 10251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } attempt: 2
I20260928 02:55:23.312966 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.312666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48526)
0928 02:55:23.312820 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.312887 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:55:23.313139 10258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45743,127.19.139.189:37399,127.19.139.188:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.189.keytab
--principal=kudu/127.19.139.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.189:37399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.139.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:23.313736 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.313514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48532)
0928 02:55:23.313627 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.313677 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:55:23.313866 10251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.189:37399: Network error: Client connection negotiation failed: client connection to 127.19.139.189:37399: connect: Connection refused (error 111)
I20260928 02:55:23.314332 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:23.315362 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:23.323764 10266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:23.323884 10263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:23.323974 10258 server_base.cc:1034] running on GCE node
W20260928 02:55:23.323760 10264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:23.324636 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:23.325809 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:23.326972 10258 hybrid_clock.cc:648] HybridClock initialized: now 1790564123326943 us; error 47 us; skew 500 ppm
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:23.329957 10258 init.cc:377] Logged in from keytab as kudu/127.19.139.189@KRBTEST.COM (short username kudu)
I20260928 02:55:23.330953 10258 webserver.cc:460] Webserver started at http://127.19.139.189:46465/ using document root <none> and password file <none>
I20260928 02:55:23.331532 10258 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:23.331605 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:23.331832 10258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:23.333984 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data/instance:
uuid: "9d7dd812e79740ab957e268ec458ccf4"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "46d38345dff7f48c601770233e74e377"
server_key_iv: "2784bd388a9d4bc912a745fa131b1613"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.334538 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal/instance:
uuid: "9d7dd812e79740ab957e268ec458ccf4"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "46d38345dff7f48c601770233e74e377"
server_key_iv: "2784bd388a9d4bc912a745fa131b1613"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.338529 10258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:55:23.341320 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:23.342689 10258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:23.342865 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal
uuid: "9d7dd812e79740ab957e268ec458ccf4"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "46d38345dff7f48c601770233e74e377"
server_key_iv: "2784bd388a9d4bc912a745fa131b1613"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.342983 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:23.357620 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:23.360988 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:23.361199 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:23.367862 10258 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.189:37399
I20260928 02:55:23.368132 10325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.189:37399 every 8 connection(s)
I20260928 02:55:23.368888 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/data/info.pb
I20260928 02:55:23.371584 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:23.375290 10326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:23.378530 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 10258
I20260928 02:55:23.378634 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-1/wal/instance
I20260928 02:55:23.378886 20014 external_mini_cluster.cc:1428] Setting key 6cf9a96ff5dddea64a3d5a09145ec95d
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.139.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.139.188@KRBTEST.COM" created.
I20260928 02:55:23.389114 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.377270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.19.139.190:45743)
0928 02:55:23.377414 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.377418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.378238 (+ 820us) server_negotiation.cc:407] Connection header received
0928 02:55:23.379198 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.379202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.379270 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.379356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.380584 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.381369 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.382495 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.382727 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.382819 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.385551 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.385554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.385556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.385583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.386957 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.387471 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.387474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.387475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.387557 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.387880 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.387883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.387884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.388089 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.388193 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.388810 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.388950 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:55:23.389950 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.377644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45743 (local address 127.0.0.1:50246)
0928 02:55:23.378087 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.378129 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.378991 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.379506 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.379513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.379906 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.380433 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.380447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.381483 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.381486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.382368 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.382376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.382910 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.382916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.383073 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.383623 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.383640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.385369 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.387089 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.387096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.387108 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.387375 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.387627 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.387629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.387630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.387759 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.388211 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.388218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.388510 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.388722 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.389065 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
I20260928 02:55:23.391439 10326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:23.403199 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.391980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48538)
0928 02:55:23.392139 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.392177 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.392249 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.392812 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.392815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.392983 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.393098 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.393104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.394091 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.394095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.394938 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.394949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.395563 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.395576 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.395679 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.396140 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.396174 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.399540 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.401494 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.401498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.401501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.401724 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.402051 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.402056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.402057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.402160 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.402758 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.402761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.402851 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.402948 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.403067 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:55:23.403393 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.392065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.19.139.189:37399)
0928 02:55:23.392407 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.392417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.392435 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:55:23.392468 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.392472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.392646 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.392808 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.393207 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.393971 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.395056 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.395449 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.395529 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.399701 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.399714 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.399727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.399752 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.401369 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.401824 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.401830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.401831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.401917 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.402286 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.402289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.402290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.402605 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.402777 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.403062 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.403192 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.139.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab.
Entry for principal kudu/127.19.139.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab.
I20260928 02:55:23.405030 10251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } attempt: 3
I20260928 02:55:23.405412 20014 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
/tmp/dist-test-taskAxMa1R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:42713
--webserver_interface=127.19.139.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab
--principal=kudu/127.19.139.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.139.148:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.139.190:45743,127.19.139.189:37399,127.19.139.188:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:23.407096 10326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:23.407852 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.407559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52468)
0928 02:55:23.407684 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.407770 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:55:23.408036 10275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
I20260928 02:55:23.409045 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.408693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52472)
0928 02:55:23.408910 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.408962 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
W20260928 02:55:23.409309 10326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:42713: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:23.414114 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.405520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.19.139.189:37399)
0928 02:55:23.405629 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.405633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.405665 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:55:23.405775 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.405778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.405822 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.405919 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.406361 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.407088 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.408054 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.408964 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.409068 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.411128 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.411132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.411134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.411155 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.412603 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.413043 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.413046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.413047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.413116 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.413379 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.413381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.413382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.413539 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.413608 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.413902 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.414013 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:55:23.414155 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.405488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48540)
0928 02:55:23.405611 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.405625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.405705 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.405949 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.405952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.406113 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.406243 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.406249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.407204 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.407208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.407946 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.407958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.409093 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.409097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.409217 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.409689 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.409703 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.410982 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.412700 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.412704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.412706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.412954 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.413195 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.413198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.413200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.413293 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.413620 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.413622 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.413731 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.413839 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.413983 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:55:23.415050 10251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:23.415834 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.415543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52478)
0928 02:55:23.415670 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.415727 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:55:23.416671 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.416418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52490)
0928 02:55:23.416567 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.416615 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 02:55:23.416785 10251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:42713: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
I20260928 02:55:23.447947 10251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } attempt: 1
I20260928 02:55:23.448936 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.448550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52496)
0928 02:55:23.448727 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.448831 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:55:23.449707 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.449485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52508)
0928 02:55:23.449594 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.449648 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:55:23.449860 10251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:42713: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
I20260928 02:55:23.458596 10326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } attempt: 1
I20260928 02:55:23.459435 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.459144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52524)
0928 02:55:23.459298 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.459359 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 02:55:23.460105 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.459898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52526)
0928 02:55:23.460000 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.460046 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:55:23.460289 10326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:42713: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
W20260928 02:55:23.517925 10332 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:55:23.518213 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:23.518236 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:23.518296 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:23.522374 10332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:23.522425 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:23.522444 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:23.522459 10332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:23.522475 10332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:23.525012 10251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } attempt: 2
I20260928 02:55:23.525822 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.525516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52542)
0928 02:55:23.525667 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.525737 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:55:23.526525 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.526330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52548)
0928 02:55:23.526418 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.526466 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:55:23.526635 10251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:42713: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
I20260928 02:55:23.528343 10332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.139.148:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.139.190:45743,127.19.139.189:37399,127.19.139.188:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/krb5kdc/kudu_127.19.139.188.keytab
--principal=kudu/127.19.139.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.139.188:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.139.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:55:23.528465 10326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } attempt: 2
I20260928 02:55:23.529094 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.528852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52554)
0928 02:55:23.528973 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.529031 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:55:23.529500 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:23.529870 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.529577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52570)
0928 02:55:23.529761 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.529813 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:55:23.529980 10326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.139.188:42713: Network error: Client connection negotiation failed: client connection to 127.19.139.188:42713: connect: Connection refused (error 111)
I20260928 02:55:23.530596 10332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:23.539089 10339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:23.539083 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:23.539083 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:23.539299 10332 server_base.cc:1034] running on GCE node
I20260928 02:55:23.539969 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:23.541098 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:23.542251 10332 hybrid_clock.cc:648] HybridClock initialized: now 1790564123542233 us; error 31 us; skew 500 ppm
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:23.545083 10332 init.cc:377] Logged in from keytab as kudu/127.19.139.188@KRBTEST.COM (short username kudu)
I20260928 02:55:23.546280 10332 webserver.cc:460] Webserver started at http://127.19.139.188:46667/ using document root <none> and password file <none>
I20260928 02:55:23.546952 10332 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:23.547031 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:23.547245 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:23.549890 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data/instance:
uuid: "edaf1076716e47c1a10e79d6a203a0e9"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "1718719736cde0ba97f23bc85576066f"
server_key_iv: "6976f10e16d9359d7b57e76bf1063508"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.550550 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal/instance:
uuid: "edaf1076716e47c1a10e79d6a203a0e9"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "1718719736cde0ba97f23bc85576066f"
server_key_iv: "6976f10e16d9359d7b57e76bf1063508"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.556842 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:55:23.559440 10349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:23.560827 10332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:23.560951 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data,/tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal
uuid: "edaf1076716e47c1a10e79d6a203a0e9"
format_stamp: "Formatted at 2026-09-28 02:55:23 on dist-test-slave-5vc0"
server_key: "1718719736cde0ba97f23bc85576066f"
server_key_iv: "6976f10e16d9359d7b57e76bf1063508"
server_key_version: "encryptionkey@0"
I20260928 02:55:23.561057 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:23.583115 10332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:23.586504 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:23.586705 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:23.593473 10332 rpc_server.cc:307] RPC server started. Bound to: 127.19.139.188:42713
I20260928 02:55:23.593727 10401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.139.188:42713 every 8 connection(s)
I20260928 02:55:23.594417 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/data/info.pb
I20260928 02:55:23.597318 10402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:23.601189 10402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:23.603298 20014 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu as pid 10332
I20260928 02:55:23.603435 20014 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAxMa1R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563845681472-20014-0/minicluster-data/master-2/wal/instance
I20260928 02:55:23.603708 20014 external_mini_cluster.cc:1428] Setting key 3d325bbd1ce7ca90bdd811e27f5c2c45
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:55:23.616355 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.603141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.19.139.190:45743)
0928 02:55:23.603323 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.603328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.604184 (+ 856us) server_negotiation.cc:407] Connection header received
0928 02:55:23.605566 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.605570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.605625 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.605726 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.606845 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.607653 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.609124 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.609480 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.609590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.612515 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.612521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.612523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.612560 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.614033 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.614642 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.614646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.614647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.614740 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.615064 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.615067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.615068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.615293 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.615390 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.616027 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.616194 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:55:23.617431 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.603495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.190:45743 (local address 127.0.0.1:50250)
0928 02:55:23.603992 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.604045 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.605350 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.605838 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.605848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.606311 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.606673 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.606694 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.607829 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.607832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.608998 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.609009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.609644 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.609647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.609840 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.610548 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.610567 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.612315 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.614156 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.614164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.614176 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.614549 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.614824 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.614826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.614827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.614969 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.615393 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.615399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.615763 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.615934 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.616516 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":241,"spinlock_wait_cycles":1792,"thread_start_us":124,"threads_started":1}
I20260928 02:55:23.617431 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.604374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.19.139.188:42713)
0928 02:55:23.604664 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.604675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.604711 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:55:23.605386 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.605414 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.605666 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.605898 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.606852 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.608028 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.609187 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.609753 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.609847 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.612898 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.612905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.612918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.612941 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.614591 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.615037 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.615043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.615045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.615129 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.615381 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.615384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.615386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.615725 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.615924 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.616087 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.616309 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":134,"threads_started":1}
I20260928 02:55:23.618691 10402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:23.621119 10326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } attempt: 3
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:23.625779 10251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } attempt: 3
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.190@KRBTEST.COM
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.189@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:55:23.628311 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.619319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.19.139.189:37399)
0928 02:55:23.619455 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.619460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.619491 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:55:23.619582 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.619585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.619626 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.619684 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.620205 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.620850 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.621706 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.621930 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.622019 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.624232 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.624237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.624239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.624267 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.625907 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.626617 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.626620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.626622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.626710 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.627352 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.627356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.627358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.627573 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.627640 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.627990 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.628106 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:55:23.628947 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.619199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.189:37399 (local address 127.0.0.1:48556)
0928 02:55:23.619435 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.619446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.619505 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.619716 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.619718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.619877 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.620019 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.620028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.620959 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.620962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.621604 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.621613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.622137 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.622140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.622241 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.622726 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.622738 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.624068 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.626026 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.626029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.626031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.626268 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.627150 (+ 882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.627153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.627155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.627247 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.627704 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.627707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.627797 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.628497 (+ 700us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.628625 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 02:55:23.629603 10402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:23.630858 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.620052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.19.139.190:45743)
0928 02:55:23.620177 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.620181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.620250 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:55:23.620346 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.620349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.620392 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.620479 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.621001 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.622155 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.623272 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.623542 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.623597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.626527 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.626531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.626534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.626560 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.628502 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.628985 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.628989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.628990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.629077 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.629417 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.629420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.629422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.629627 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.629871 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.630590 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.630704 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.190@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:55:23.632462 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.621483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52592)
0928 02:55:23.621643 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.621656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.621747 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.621974 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.621977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.622175 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.622285 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.622291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.623392 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.623396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.624229 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.624239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.624840 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.624845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.624944 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.625569 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.625584 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.627026 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.630302 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.630305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.630307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.630564 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.631208 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.631211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.631213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.631308 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.631876 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.631879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.631975 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.632051 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.632295 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:55:23.633260 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.621568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.19.139.188:42713)
0928 02:55:23.621665 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.621669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.621704 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:55:23.621833 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.621836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.621877 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.621940 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.622542 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.623283 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.624316 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.624653 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.624797 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.627146 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.627150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.627152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.627202 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.630187 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.630711 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.630714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.630716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.630791 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.631539 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.631542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.631562 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.631760 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.632908 (+ 1148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.632982 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.633096 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":4864}
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564123, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.139.188@KRBTEST.COM for kudu/127.19.139.188@KRBTEST.COM
I20260928 02:55:23.636559 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: Bootstrap starting.
Sep 28 02:55:23 dist-test-slave-5vc0 krb5kdc[10170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.139.189@KRBTEST.COM
I20260928 02:55:23.639462 10326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:23.639560 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.630105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52604)
0928 02:55:23.630514 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.630526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.630585 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.631817 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.631819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.631971 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.632053 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.632060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.633013 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.633016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.633766 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.633775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.634391 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.634395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.634464 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.634843 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.634857 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.636102 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.638097 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.638101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.638103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.638313 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.638580 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.638583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.638585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.638674 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.639076 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.639079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.639169 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.639280 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.639400 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350}
I20260928 02:55:23.640022 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.630274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.19.139.188:42713)
0928 02:55:23.631544 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.631555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.631569 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:55:23.631599 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.631601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.631641 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.631710 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.632177 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.632916 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.633868 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.634204 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.634287 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.636275 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.636279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.636281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.636304 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.638004 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.638420 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.638424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.638426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.638497 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.638759 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.638763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.638764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.638978 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.639074 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.639348 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.639462 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":116,"threads_started":1}
I20260928 02:55:23.640343 10326 log.cc:824] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:23.641040 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.632254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.19.139.189:37399)
0928 02:55:23.632352 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.632355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.632499 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:55:23.632641 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.632644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.632682 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.632727 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.633520 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.634223 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.635202 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.635494 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.635590 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.637844 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.637847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.637849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.637869 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.639282 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.639722 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.639726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.639728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.639805 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.640095 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.640099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.640101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.640337 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.640492 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.640811 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.640913 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:55:23.642104 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.626245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.139.188:42713 (local address 127.0.0.1:52600)
0928 02:55:23.626374 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:23.626389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:23.626459 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:23.627457 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:23.627460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:23.627650 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:23.627769 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.627777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.629303 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.629307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.630022 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:23.630033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.630611 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.630615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:23.630701 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.631073 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:23.631086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:23.632435 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:23.640314 (+ 7879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.640317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.640319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.640572 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.640902 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:23.640905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:23.640907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:23.641005 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:23.641635 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:23.641638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:23.641737 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:55:23.641830 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:55:23.641963 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:55:23.642963 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.626830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.19.139.188:42713)
0928 02:55:23.627225 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.627234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.627245 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:55:23.627271 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.627274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.627315 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.627576 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.627922 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.629196 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.630148 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.630489 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.630858 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.632580 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.632584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.632586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.632608 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.634168 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.640692 (+ 6524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.640694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.640696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.640777 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.641110 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.641113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.641115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.641318 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.641412 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.642707 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.642816 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":140,"threads_started":1}
I20260928 02:55:23.643033 10402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: Bootstrap starting.
I20260928 02:55:23.643719 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: No bootstrap required, opened a new log
I20260928 02:55:23.645447 10402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:23.645726 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068: Bootstrap starting.
I20260928 02:55:23.646188 10402 log.cc:824] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:23.648023 10402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: No bootstrap required, opened a new log
I20260928 02:55:23.648132 10326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.648331 10251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:23.648396 10326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7dd812e79740ab957e268ec458ccf4, State: Initialized, Role: FOLLOWER
I20260928 02:55:23.648733 20014 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:23.649183 10251 log.cc:824] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:23.649660 10326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.651049 10326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:23.649968 10225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f6a1e20177940d0a9404f54c3dd0068"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50258
I20260928 02:55:23.651878 10402 raft_consensus.cc:348] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.652110 10402 raft_consensus.cc:729] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edaf1076716e47c1a10e79d6a203a0e9, State: Initialized, Role: FOLLOWER
I20260928 02:55:23.652426 10402 consensus_queue.cc:260] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.653416 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068: No bootstrap required, opened a new log
I20260928 02:55:23.653666 10412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.653780 10412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:23.654012 10419 sys_catalog.cc:455] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.654150 10419 sys_catalog.cc:458] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:55:23.656289 10424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:23.656414 10424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:23.656630 10402 sys_catalog.cc:564] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:23.656679 10251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.656883 10251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6a1e20177940d0a9404f54c3dd0068, State: Initialized, Role: FOLLOWER
I20260928 02:55:23.657199 10251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.654574 10258 master_runner.cc:186] Error getting master registration for 127.19.139.190:45743: OK, instance_id { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" instance_seqno: 1790564123160677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:55:23.657886 10258 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:23.657727 10428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.658133 10428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:23.658496 10251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:23.658562 10225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:55:23.658676 10225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:23.659938 10225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.660897 10225 leader_election.cc:288] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [CANDIDATE]: Term 1 election: Requested vote from peers 9d7dd812e79740ab957e268ec458ccf4 (127.19.139.189:37399), edaf1076716e47c1a10e79d6a203a0e9 (127.19.139.188:42713)
W20260928 02:55:23.661705 10437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:23.661835 10437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:23.662020 10300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f6a1e20177940d0a9404f54c3dd0068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9d7dd812e79740ab957e268ec458ccf4"
I20260928 02:55:23.662232 10300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:23.659894 10332 master_runner.cc:186] Error getting master registration for 127.19.139.190:45743: OK, instance_id { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" instance_seqno: 1790564123160677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:55:23.663147 10332 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:23.663280 10300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6a1e20177940d0a9404f54c3dd0068 in term 1.
I20260928 02:55:23.663692 10202 leader_election.cc:302] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7dd812e79740ab957e268ec458ccf4, 9f6a1e20177940d0a9404f54c3dd0068; no voters:
I20260928 02:55:23.663705 10376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f6a1e20177940d0a9404f54c3dd0068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "edaf1076716e47c1a10e79d6a203a0e9"
I20260928 02:55:23.663944 10376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:23.663980 10428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:23.664208 10428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [term 1 LEADER]: Becoming Leader. State: Replica: 9f6a1e20177940d0a9404f54c3dd0068, State: Running, Role: LEADER
I20260928 02:55:23.664553 10428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } }
I20260928 02:55:23.666389 10428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6a1e20177940d0a9404f54c3dd0068. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.666484 10428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:23.666970 10376 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6a1e20177940d0a9404f54c3dd0068 in term 1.
I20260928 02:55:23.667171 10455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:23.674152 10455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:23.686174 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.669849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.19.139.188:42713)
0928 02:55:23.669988 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.669992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.675746 (+ 5754us) server_negotiation.cc:407] Connection header received
0928 02:55:23.675915 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.675918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.675967 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.676014 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.676988 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.677702 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.678813 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.679159 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.679229 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.681642 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.681646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.681649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.681677 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.683647 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.684818 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.684821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.684823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.684917 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.685202 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.685206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.685208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.685430 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.685552 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.685878 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.686007 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:55:23.687819 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.668350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.19.139.190:45743)
0928 02:55:23.668474 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.668478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.671783 (+ 3305us) server_negotiation.cc:407] Connection header received
0928 02:55:23.671898 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.671901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.671981 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.672273 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.677494 (+ 5221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.678362 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.679513 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.680141 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.680277 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.683819 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.683822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.683825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.683852 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.685662 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.686105 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.686109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.686110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.686193 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.686861 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.686864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.686866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.687074 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.687179 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.687537 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.687669 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:55:23.690510 10376 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [term 1 FOLLOWER]: Refusing update from remote peer 9f6a1e20177940d0a9404f54c3dd0068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:23.690594 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.668059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.19.139.189:37399)
0928 02:55:23.668216 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.668221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.670943 (+ 2722us) server_negotiation.cc:407] Connection header received
0928 02:55:23.671310 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.671314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.671396 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.671463 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.672767 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.673511 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.674740 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.675067 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.675141 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.680933 (+ 5792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.680937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.680939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.680970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.685488 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.689225 (+ 3737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.689230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.689232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.689338 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.689646 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.689649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.689650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.689887 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.690005 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.690331 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.690450 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:55:23.690729 10300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [term 1 FOLLOWER]: Refusing update from remote peer 9f6a1e20177940d0a9404f54c3dd0068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:23.691810 10442 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:23.692981 10442 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:23.697875 10412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6a1e20177940d0a9404f54c3dd0068. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.698038 10412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:23.698487 10419 sys_catalog.cc:455] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f6a1e20177940d0a9404f54c3dd0068. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.698642 10419 sys_catalog.cc:458] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:23.705615 10442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.705761 10442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:23.707034 10412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.707156 10412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:23.709789 10461 mvcc.cc:205] Tried to move back new op lower bound from 7334150650599473152 to 7334150650533584896. Current Snapshot: MvccSnapshot[applied={T|T < 7334150650599473152}]
I20260928 02:55:23.710011 10462 mvcc.cc:205] Tried to move back new op lower bound from 7334150650599473152 to 7334150650533584896. Current Snapshot: MvccSnapshot[applied={T|T < 7334150650599473152}]
I20260928 02:55:23.714474 10428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.714605 10428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:23.715801 10419 sys_catalog.cc:455] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f6a1e20177940d0a9404f54c3dd0068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6a1e20177940d0a9404f54c3dd0068" member_type: VOTER last_known_addr { host: "127.19.139.190" port: 45743 } } peers { permanent_uuid: "9d7dd812e79740ab957e268ec458ccf4" member_type: VOTER last_known_addr { host: "127.19.139.189" port: 37399 } } peers { permanent_uuid: "edaf1076716e47c1a10e79d6a203a0e9" member_type: VOTER last_known_addr { host: "127.19.139.188" port: 42713 } } }
I20260928 02:55:23.715921 10419 sys_catalog.cc:458] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:23.716413 10455 catalog_manager.cc:1347] Generated new cluster ID: 740c86d32d134ebfb235b1b76f9da1ce
I20260928 02:55:23.716480 10455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:23.716681 10463 mvcc.cc:205] Tried to move back new op lower bound from 7334150650599473152 to 7334150650533584896. Current Snapshot: MvccSnapshot[applied={T|T < 7334150650599473152}]
I20260928 02:55:23.734908 10455 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:23.736804 10455 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:23.760668 10455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f6a1e20177940d0a9404f54c3dd0068: Generated new TSK 0
I20260928 02:55:23.766793 10455 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:55:23.800447 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.797099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.19.139.189:37399)
0928 02:55:23.797347 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.797352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.799278 (+ 1926us) server_negotiation.cc:407] Connection header received
0928 02:55:23.799496 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.799501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.799552 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.799604 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.800292 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48580: BlockingRecv error: recv got EOF from 127.0.0.1:48580 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260928 02:55:23.802093 10407 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.798711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.19.139.188:42713)
0928 02:55:23.798822 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.798826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.801200 (+ 2374us) server_negotiation.cc:407] Connection header received
0928 02:55:23.801378 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.801382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.801431 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.801479 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.801966 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52626: BlockingRecv error: recv got EOF from 127.0.0.1:52626 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:55:23.804989 10255 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.798084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.19.139.190:45743)
0928 02:55:23.798192 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.798197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.800954 (+ 2757us) server_negotiation.cc:407] Connection header received
0928 02:55:23.801065 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.801068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.801121 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.801197 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.804860 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50268: BlockingRecv error: recv got EOF from 127.0.0.1:50268 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 02:55:23.813714 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.807987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.19.139.189:37399)
0928 02:55:23.808101 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.808105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.812533 (+ 4428us) server_negotiation.cc:407] Connection header received
0928 02:55:23.812796 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.812800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.812889 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.812945 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.813587 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48590: BlockingRecv error: recv got EOF from 127.0.0.1:48590 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:55:23.818988 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.817363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.19.139.189:37399)
0928 02:55:23.817477 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.817481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.818033 (+ 552us) server_negotiation.cc:407] Connection header received
0928 02:55:23.818224 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.818228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.818279 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.818341 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.818870 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48604: BlockingRecv error: recv got EOF from 127.0.0.1:48604 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:55:23.824542 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.822980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.19.139.189:37399)
0928 02:55:23.823086 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.823090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.823604 (+ 514us) server_negotiation.cc:407] Connection header received
0928 02:55:23.823786 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.823789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.823843 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.823894 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.824429 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48608: BlockingRecv error: recv got EOF from 127.0.0.1:48608 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 02:55:23.832643 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.830740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.19.139.189:37399)
0928 02:55:23.830892 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.830898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.831579 (+ 681us) server_negotiation.cc:407] Connection header received
0928 02:55:23.831816 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.831820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.831878 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.831930 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.832513 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48622: BlockingRecv error: recv got EOF from 127.0.0.1:48622 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 02:55:23.845001 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.843090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.19.139.189:37399)
0928 02:55:23.843223 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.843227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.843878 (+ 651us) server_negotiation.cc:407] Connection header received
0928 02:55:23.844097 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.844100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.844206 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.844264 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.844874 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48628: BlockingRecv error: recv got EOF from 127.0.0.1:48628 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:55:23.866446 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.864653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.19.139.189:37399)
0928 02:55:23.864800 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.864804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.865396 (+ 592us) server_negotiation.cc:407] Connection header received
0928 02:55:23.865633 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.865636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.865697 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.865785 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.866319 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48644: BlockingRecv error: recv got EOF from 127.0.0.1:48644 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 02:55:23.902457 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.901072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.19.139.189:37399)
0928 02:55:23.901202 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.901207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.901495 (+ 288us) server_negotiation.cc:407] Connection header received
0928 02:55:23.901628 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.901631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.901697 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.901795 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.902329 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48646: BlockingRecv error: recv got EOF from 127.0.0.1:48646 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:55:23.969898 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:23.968653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.19.139.189:37399)
0928 02:55:23.968797 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.968802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.968977 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:55:23.969133 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.969136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.969187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.969266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.969765 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48658: BlockingRecv error: recv got EOF from 127.0.0.1:48658 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:55:24.101541 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:24.100349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.19.139.189:37399)
0928 02:55:24.100483 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.100488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.100678 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:55:24.100828 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.100833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.100885 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.100955 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:24.101441 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48662: BlockingRecv error: recv got EOF from 127.0.0.1:48662 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 02:55:24.360765 10331 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:55:24.359679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.19.139.189:37399)
0928 02:55:24.359825 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.359831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.359884 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:55:24.360043 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.360046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.360092 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.360193 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:24.360640 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48670: BlockingRecv error: recv got EOF from 127.0.0.1:48670 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:55:24.657799 10424 catalog_manager.cc:1259] Loaded cluster ID: 740c86d32d134ebfb235b1b76f9da1ce
I20260928 02:55:24.657905 10424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: loading cluster ID for follower catalog manager: success
I20260928 02:55:24.659579 10424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: acquiring CA information for follower catalog manager: success
I20260928 02:55:24.660766 10424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d7dd812e79740ab957e268ec458ccf4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:55:24.662920 10437 catalog_manager.cc:1259] Loaded cluster ID: 740c86d32d134ebfb235b1b76f9da1ce
I20260928 02:55:24.662976 10437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: loading cluster ID for follower catalog manager: success
I20260928 02:55:24.664645 10437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: acquiring CA information for follower catalog manager: success
I20260928 02:55:24.665676 10437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edaf1076716e47c1a10e79d6a203a0e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:55:24.878841 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:24.874948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.19.139.189:37399)
0928 02:55:24.875203 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.875206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.875221 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:55:24.875328 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.875331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.875462 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.875562 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:24.875907 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.876483 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.877386 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.877608 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.877676 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:24.878022 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:24.878478 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:24.878567 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:24.878657 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 02:55:24.878713 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260928 02:55:24.885941 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:24.882021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.19.139.188:42713)
0928 02:55:24.882237 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.882240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.882253 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:55:24.882332 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.882334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.882446 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.882544 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:24.882829 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.883398 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.884325 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.884553 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.884639 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:24.885028 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:24.885478 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:24.885572 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:24.885713 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:55:24.885777 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260928 02:55:24.892627 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:24.889002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.19.139.189:37399)
0928 02:55:24.889130 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.889134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.889584 (+ 450us) server_negotiation.cc:407] Connection header received
0928 02:55:24.889681 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.889684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.889736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.889805 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:24.890024 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.890605 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.891589 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.891796 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.891889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:24.892182 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:24.892268 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:24.892333 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:24.892431 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:55:24.892486 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:55:24.893764 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:24.889363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.19.139.188:42713)
0928 02:55:24.889464 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.889468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.889614 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:55:24.889780 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.889783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.889831 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.889893 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:24.890269 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.891045 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.892582 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.892875 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.892949 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:24.893272 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:24.893376 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:24.893442 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:24.893577 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:55:24.893636 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:55:24.893980 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:24.889111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.19.139.190:45743)
0928 02:55:24.889369 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:24.889373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:24.889490 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:55:24.889592 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:24.889595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:24.889771 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:24.889899 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:24.890226 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.891046 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.892224 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:24.892563 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:24.892655 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:24.893189 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:24.893563 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:24.893636 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:24.893777 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:55:24.893841 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260928 02:55:24.897289 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 10183
I20260928 02:55:24.902434 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 10258
I20260928 02:55:24.907454 20014 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAxMa1R/build/debug/bin/kudu with pid 10332
2026-09-28T02:55:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2998 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5993 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279218 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST